Process of notebook:

  1. Problem Definition
  2. Data
  3. Evaluate
  4. Features

1. Problem Definition¶

How can I predict my body weight (in kgs), in addition to my waist circumference (in cm), just by using the calories I've consumed, in addition to my macronutrients (in grams) and it's percentage in calories?

2. Data¶

We're working with semi-structured excel sheet data. The data I recorded was both weight and waist measurement, in the morning and in the night. But I've slowly transitioned to only measuring in the morning instead. And I didn't track my calories and macros until a little later into my excel recordings. So there will be a number of data cleaning.

We'll be working with date-time data, as the weight and waist data is associated to its date-time, in addition to the calories and macro details.

There is no established train or test dataset. So we'll split the roughly 3800 cels into these 2 sets of dataframes.

3. Evaluation¶

We will evaluate by how well the ML program could predict my weight, only by the calories and macro data, in addition to using my weight and waist from the previous days as reference for its prediction.

4. Features¶

We will only have 7 different features to given, in order to predict my weight and waist. But we will also need to factor in my weight and waist from the previous day/s as well, as my current body weight, also has an affect on the weight I either gain or lose on the following days.

We can form 2 extra features to help us. The positive and negative difference between the weight and waist results from the previous day, to current day.

In [1]:
# Timestamp - to make recording on when this project was first made
import datetime

print(f'Start of notebook run: {datetime.datetime.now()}')

Start of notebook run: 2025-04-15 15:32:05.169992
In [2]:
# import analysis tools
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
In [3]:
# Import excel file
!pip install openpyxl
data = pd.read_excel('Weight watching data.xlsx')

data.head()

Requirement already satisfied: openpyxl in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages (3.1.5)
Requirement already satisfied: et-xmlfile in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages (from openpyxl) (2.0.0)

WARNING: You are using pip version 22.0.4; however, version 25.0.1 is available.
You should consider upgrading via the 'C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip' command.
Out[3]:
Date Notes Weight (kg) Unnamed: 3 Waist (cm) Unnamed: 5 BMI Unnamed: 7 Unnamed: 8 Unnamed: 9 ... Unnamed: 58 Unnamed: 59 Unnamed: 60 Unnamed: 61 Unnamed: 62 Unnamed: 63 Unnamed: 64 Unnamed: 65 Unnamed: 66 Unnamed: 67
0 Start - Aug 20th 2019 NaN 53.7 NaN 66.0 NaN 17.736821 NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 NaN NaN 53.6 NaN 64.5 NaN 17.703792 NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 NaN NaN 53.7 NaN 64.0 NaN 17.736821 NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 NaN NaN 53.2 NaN 68.2 NaN 17.571674 NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 NaN NaN 52.9 NaN 66.2 NaN 17.472586 NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 68 columns

In [4]:
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30637 entries, 0 to 30636
Data columns (total 68 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   Date                 30172 non-null  object 
 1   Notes                15 non-null     object 
 2   Weight (kg)          3853 non-null   float64
 3   Unnamed: 3           2 non-null      object 
 4   Waist (cm)           3853 non-null   float64
 5   Unnamed: 5           0 non-null      float64
 6   BMI                  3853 non-null   float64
 7   Unnamed: 7           0 non-null      float64
 8   Unnamed: 8           74 non-null     object 
 9   Unnamed: 9           1 non-null      object 
 10  Unnamed: 10          8490 non-null   object 
 11  Unnamed: 11          0 non-null      float64
 12  Unnamed: 12          1959 non-null   object 
 13  Unnamed: 13          0 non-null      float64
 14  Unnamed: 14          3916 non-null   object 
 15  Unnamed: 15          0 non-null      float64
 16  Fast length (hours)  4156 non-null   object 
 17  Unnamed: 17          1 non-null      object 
 18  Fast avg. 7 (hours)  4150 non-null   object 
 19  Unnamed: 19          0 non-null      float64
 20  Height               1690 non-null   object 
 21  Unnamed: 21          0 non-null      float64
 22  Protein (g)          1690 non-null   object 
 23  Unnamed: 23          0 non-null      float64
 24  Fat (g)              1689 non-null   object 
 25  Unnamed: 25          0 non-null      float64
 26  Carbohydrate %       1689 non-null   object 
 27  Unnamed: 27          0 non-null      float64
 28  Protein %            1689 non-null   object 
 29  Unnamed: 29          0 non-null      float64
 30  Fat %                1689 non-null   object 
 31  Unnamed: 31          0 non-null      float64
 32  Macro score          1689 non-null   object 
 33  Unnamed: 33          0 non-null      float64
 34  Score*Log. Kcal      1689 non-null   object 
 35  Unnamed: 35          0 non-null      float64
 36  Weight (kg).1        3916 non-null   object 
 37  Unnamed: 37          0 non-null      float64
 38  Waist (cm).1         3916 non-null   object 
 39  Unnamed: 39          0 non-null      float64
 40  Unnamed: 40          4522 non-null   object 
 41  Unnamed: 41          0 non-null      float64
 42  Unnamed: 42          2013 non-null   object 
 43  Unnamed: 43          0 non-null      float64
 44  Unnamed: 44          221 non-null    object 
 45  Unnamed: 45          0 non-null      float64
 46  Unnamed: 46          0 non-null      float64
 47  Unnamed: 47          0 non-null      float64
 48  Unnamed: 48          0 non-null      float64
 49  Unnamed: 49          0 non-null      float64
 50  Unnamed: 50          0 non-null      float64
 51  Unnamed: 51          0 non-null      float64
 52  Unnamed: 52          0 non-null      float64
 53  Unnamed: 53          0 non-null      float64
 54  Unnamed: 54          0 non-null      float64
 55  Unnamed: 55          0 non-null      float64
 56  Unnamed: 56          0 non-null      float64
 57  Unnamed: 57          0 non-null      float64
 58  Unnamed: 58          0 non-null      float64
 59  Unnamed: 59          0 non-null      float64
 60  Unnamed: 60          0 non-null      float64
 61  Unnamed: 61          0 non-null      float64
 62  Unnamed: 62          0 non-null      float64
 63  Unnamed: 63          0 non-null      float64
 64  Unnamed: 64          0 non-null      float64
 65  Unnamed: 65          0 non-null      float64
 66  Unnamed: 66          0 non-null      float64
 67  Unnamed: 67          1 non-null      object 
dtypes: float64(43), object(25)
memory usage: 15.9+ MB

Unfortunately, there's a lot of NaN values in my dataset. Fortunately, I'm familiar with what my data set is.

If I've filled in any type of value in my excel sheet, then its a valid point of my data. Any data with missing values, will not be counted, as I have not entered any value in the cels.

The columns with few to no values, are either used for communication purposes, or column separation to make my data input/viewing more easier on the eyes respectfully. Therefore can be removed, due to no particular use to our goals of this project.

In [5]:
# Remove columns with 1000 or fewer variables within it

df = data.loc[:, data.count() > 1000] # .loc = locating something from our data, representing = [row:column]
                                      # aka counting 1000 or more values in our column, and locating them into our new variable, df
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30637 entries, 0 to 30636
Data columns (total 21 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   Date                 30172 non-null  object 
 1   Weight (kg)          3853 non-null   float64
 2   Waist (cm)           3853 non-null   float64
 3   BMI                  3853 non-null   float64
 4   Unnamed: 10          8490 non-null   object 
 5   Unnamed: 12          1959 non-null   object 
 6   Unnamed: 14          3916 non-null   object 
 7   Fast length (hours)  4156 non-null   object 
 8   Fast avg. 7 (hours)  4150 non-null   object 
 9   Height               1690 non-null   object 
 10  Protein (g)          1690 non-null   object 
 11  Fat (g)              1689 non-null   object 
 12  Carbohydrate %       1689 non-null   object 
 13  Protein %            1689 non-null   object 
 14  Fat %                1689 non-null   object 
 15  Macro score          1689 non-null   object 
 16  Score*Log. Kcal      1689 non-null   object 
 17  Weight (kg).1        3916 non-null   object 
 18  Waist (cm).1         3916 non-null   object 
 19  Unnamed: 40          4522 non-null   object 
 20  Unnamed: 42          2013 non-null   object 
dtypes: float64(3), object(18)
memory usage: 4.9+ MB

So now we have a range of different numbers, still with many NaN. How do we figure out what to throw or keep?

We will keep the column's data, that we'll be utilizing in our project, that has the least amount of inputted data. For our example, 1,689 is currently the lowest number of data we have that we'll be using. Which is namely our macro data. We do have a weird column named height though with 1,690 values.

Let's check out the dataframe and have a deeper dive what it displays.

In [6]:
df.head()
Out[6]:
Date Weight (kg) Waist (cm) BMI Unnamed: 10 Unnamed: 12 Unnamed: 14 Fast length (hours) Fast avg. 7 (hours) Height ... Fat (g) Carbohydrate % Protein % Fat % Macro score Score*Log. Kcal Weight (kg).1 Waist (cm).1 Unnamed: 40 Unnamed: 42
0 Start - Aug 20th 2019 53.7 66.0 17.736821 NaN NaN NaN 13 20 1.74 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 NaN 53.6 64.5 17.703792 NaN NaN NaN 26 21.142857 NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 NaN 53.7 64.0 17.736821 NaN NaN NaN 21 19.428571 NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 NaN 53.2 68.2 17.571674 NaN NaN NaN 15 19.714286 NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 NaN 52.9 66.2 17.472586 NaN NaN NaN 32 20.428571 NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 21 columns

With Height, it's likely one singular datapoint to hold the variable of my height, to use to calculate and do some feature engineering in my excel sheet. For example, my BMI data.

But protein has 1,690 values. So I wonder what extra variable is in it?

In [7]:
# Separate numeric and string values
numeric_values = pd.to_numeric(df['Protein (g)'], errors='coerce') # convert numeric values, and leave NaN for any non-numeric variables
strings = df['Protein (g)'][df['Protein (g)'].apply(lambda x: isinstance(x, str))] # filters for rows that are strings

# Sort each group independently
numeric_values_sorted = sorted(numeric_values.dropna(), reverse=True)
strings_sorted = sorted(strings)

# Combine them if needed
sorted_values = numeric_values_sorted + strings_sorted

print(sorted_values)

[374.4, 356.4, 333.6, 317.7, 313.2, 302.1, 294.7, 294.7, 279.0, 277.9, 276.6, 272.8, 270.5, 268.8, 268.6, 268.5, 265.2, 263.9, 263.9, 263.8, 263.3, 263.0, 262.4, 260.4, 259.5, 258.2, 257.9, 256.4, 254.5, 253.5, 253.3, 252.6, 252.2, 252.2, 251.1, 251.1, 250.6, 250.5, 249.5, 249.3, 247.9, 247.4, 246.0, 243.9, 243.6, 242.5, 241.8, 241.5, 239.4, 236.0, 235.9, 235.6, 235.3, 234.7, 234.5, 234.2, 233.3, 232.8, 230.9, 230.4, 229.9, 229.7, 229.5, 228.4, 227.7, 227.5, 227.1, 225.6, 225.6, 225.5, 224.9, 224.4, 224.1, 223.9, 223.6, 222.7, 222.6, 222.1, 221.4, 220.8, 220.6, 220.5, 220.1, 219.7, 219.2, 218.7, 218.4, 218.2, 217.7, 216.5, 216.4, 216.2, 216.1, 215.9, 215.9, 215.6, 215.4, 214.9, 214.6, 214.5, 214.2, 214.1, 214.0, 213.9, 213.0, 212.6, 212.5, 212.3, 212.3, 211.9, 211.7, 211.5, 211.3, 210.8, 210.7, 210.6, 210.3, 209.6, 209.4, 209.3, 208.5, 207.7, 207.6, 207.5, 207.2, 207.2, 207.1, 207.1, 207.0, 206.9, 206.8, 206.2, 205.9, 204.9, 204.4, 204.1, 203.9, 203.8, 203.4, 203.4, 202.8, 202.5, 202.4, 202.3, 201.5, 201.5, 201.1, 200.8, 200.5, 200.3, 200.1, 200.0, 199.9, 199.3, 198.1, 198.0, 197.9, 197.3, 197.3, 196.8, 196.8, 196.8, 196.2, 196.2, 196.2, 196.1, 195.6, 195.3, 195.1, 195.0, 194.9, 194.8, 194.6, 194.1, 194.0, 193.2, 193.0, 193.0, 192.1, 192.0, 191.8, 191.8, 191.7, 191.7, 191.6, 191.4, 191.4, 191.4, 191.1, 190.2, 190.1, 190.0, 190.0, 189.6, 189.3, 189.2, 189.1, 188.7, 188.5, 188.5, 188.4, 188.3, 188.2, 188.1, 187.7, 187.7, 187.4, 187.2, 187.0, 186.8, 186.3, 185.9, 185.7, 185.3, 185.1, 184.8, 184.7, 184.7, 184.5, 184.5, 184.3, 184.3, 184.3, 184.0, 184.0, 184.0, 183.6, 183.6, 183.4, 183.4, 183.4, 183.3, 183.2, 183.1, 183.0, 182.7, 182.6, 182.3, 182.1, 182.0, 181.9, 181.2, 180.6, 180.4, 180.2, 179.2, 179.2, 179.1, 179.0, 179.0, 178.8, 178.8, 178.7, 178.3, 178.0, 178.0, 177.7, 177.4, 177.3, 177.0, 177.0, 177.0, 176.8, 176.8, 176.8, 176.5, 176.4, 176.2, 175.9, 175.8, 175.5, 175.4, 175.3, 175.2, 175.0, 175.0, 175.0, 174.9, 174.7, 174.5, 174.3, 174.2, 174.1, 173.8, 173.6, 173.5, 173.5, 173.2, 172.8, 172.7, 172.2, 172.0, 171.9, 171.9, 171.4, 171.2, 171.1, 170.8, 170.6, 170.3, 170.0, 169.9, 169.9, 169.6, 169.4, 169.2, 169.2, 168.7, 168.5, 168.4, 168.3, 168.1, 167.6, 167.6, 167.4, 167.4, 167.4, 167.2, 166.8, 166.4, 166.4, 166.3, 166.1, 166.0, 165.9, 165.5, 165.4, 165.4, 165.3, 165.1, 165.1, 165.0, 164.9, 164.8, 164.8, 164.8, 164.3, 164.1, 163.9, 163.6, 163.4, 163.4, 163.4, 163.3, 163.2, 163.0, 163.0, 162.9, 162.5, 162.5, 162.3, 162.0, 161.5, 161.5, 161.5, 161.0, 160.5, 159.8, 159.7, 159.7, 159.4, 159.3, 159.2, 158.7, 158.7, 158.5, 158.5, 158.5, 158.3, 157.9, 157.6, 157.6, 157.0, 156.6, 156.3, 156.3, 156.2, 156.0, 156.0, 155.7, 155.5, 155.2, 155.1, 155.1, 154.9, 154.8, 154.7, 154.4, 154.1, 154.0, 153.9, 153.4, 152.6, 152.0, 151.7, 151.5, 151.5, 151.5, 151.2, 151.0, 151.0, 150.5, 150.4, 150.1, 149.9, 149.7, 149.2, 149.1, 148.9, 148.9, 148.6, 148.5, 148.3, 148.3, 148.3, 148.2, 148.0, 147.9, 147.8, 147.7, 147.3, 146.8, 146.8, 146.7, 146.7, 146.6, 146.5, 146.5, 146.2, 145.7, 145.5, 144.4, 144.1, 144.0, 143.7, 143.7, 143.7, 143.6, 143.3, 143.0, 142.9, 142.9, 142.6, 142.5, 142.2, 142.2, 141.9, 141.8, 141.7, 141.6, 141.3, 141.1, 140.9, 140.8, 140.4, 140.2, 140.0, 139.9, 139.5, 139.5, 139.4, 139.2, 138.9, 138.8, 138.7, 138.6, 138.4, 138.3, 137.8, 137.7, 137.2, 136.9, 136.4, 136.3, 136.1, 136.0, 135.6, 135.5, 135.0, 134.7, 134.5, 134.4, 134.2, 134.1, 134.1, 133.9, 133.6, 132.8, 132.8, 132.7, 132.7, 132.6, 132.5, 132.5, 132.1, 132.1, 131.7, 131.5, 131.0, 130.9, 130.5, 130.5, 130.4, 130.3, 130.3, 130.2, 129.6, 129.6, 129.3, 129.2, 129.0, 128.8, 128.7, 128.7, 128.4, 128.4, 128.4, 128.2, 128.1, 128.1, 128.1, 128.1, 127.8, 127.5, 127.5, 127.5, 127.4, 127.3, 127.1, 126.9, 126.7, 126.4, 126.0, 126.0, 125.8, 125.4, 125.3, 125.1, 125.0, 125.0, 124.9, 124.7, 124.2, 124.2, 124.1, 123.9, 123.1, 123.0, 123.0, 123.0, 122.6, 122.6, 122.5, 122.5, 122.4, 122.4, 122.1, 122.1, 121.1, 121.0, 120.9, 120.6, 120.3, 120.2, 120.0, 119.8, 119.3, 119.1, 119.1, 119.0, 118.8, 118.8, 118.7, 118.6, 118.5, 118.4, 118.1, 118.0, 117.9, 117.8, 117.5, 117.5, 117.5, 117.1, 117.1, 116.8, 116.7, 116.7, 116.6, 116.4, 116.3, 116.1, 116.1, 116.0, 116.0, 115.9, 115.9, 115.5, 115.5, 115.5, 115.2, 115.2, 114.9, 114.2, 114.2, 114.1, 114.1, 114.0, 113.7, 113.7, 113.7, 113.7, 113.3, 113.1, 113.1, 113.0, 112.9, 112.9, 112.8, 112.5, 112.3, 111.8, 111.7, 111.5, 111.0, 110.9, 110.7, 110.4, 110.3, 110.1, 110.0, 109.9, 109.9, 109.6, 109.6, 109.6, 109.5, 109.4, 109.3, 109.3, 109.2, 109.2, 109.1, 109.1, 109.1, 109.0, 109.0, 108.9, 108.7, 108.7, 108.7, 108.5, 108.0, 107.9, 107.6, 107.5, 107.5, 107.5, 107.2, 107.1, 107.1, 107.0, 107.0, 106.8, 106.8, 106.6, 106.4, 106.3, 106.2, 106.1, 106.0, 105.9, 105.6, 105.5, 105.4, 105.4, 105.4, 105.3, 105.2, 105.2, 105.1, 105.0, 105.0, 105.0, 104.6, 104.6, 104.5, 104.4, 104.2, 104.1, 104.0, 104.0, 104.0, 104.0, 103.7, 103.7, 103.6, 103.3, 103.0, 102.7, 102.7, 102.5, 102.5, 102.4, 102.4, 102.3, 102.1, 102.0, 102.0, 101.9, 101.8, 101.7, 101.7, 101.7, 101.6, 101.5, 101.5, 101.5, 101.4, 101.4, 101.4, 101.3, 101.3, 101.2, 101.2, 101.2, 101.1, 100.9, 100.8, 100.6, 100.5, 100.4, 100.4, 100.2, 100.1, 100.0, 100.0, 100.0, 99.9, 99.7, 99.7, 99.5, 99.5, 99.4, 99.4, 99.3, 99.0, 99.0, 99.0, 98.9, 98.6, 98.5, 98.4, 98.4, 98.3, 98.2, 98.2, 98.1, 98.0, 97.9, 97.8, 97.6, 97.3, 97.2, 97.2, 97.2, 97.2, 97.1, 97.0, 96.9, 96.8, 96.7, 96.7, 96.5, 96.4, 96.3, 96.1, 96.0, 96.0, 95.9, 95.8, 95.8, 95.8, 95.7, 95.5, 95.4, 95.4, 95.3, 95.2, 95.1, 95.1, 95.1, 94.8, 94.8, 94.5, 94.5, 94.4, 94.4, 94.4, 94.3, 94.3, 94.2, 94.1, 94.1, 93.9, 93.8, 93.8, 93.8, 93.7, 93.7, 93.6, 93.4, 93.4, 93.3, 93.1, 93.1, 93.0, 93.0, 92.9, 92.9, 92.8, 92.7, 92.7, 92.7, 92.6, 92.6, 92.6, 92.6, 92.5, 92.5, 92.5, 92.5, 92.4, 92.3, 92.3, 92.2, 92.2, 92.1, 92.0, 92.0, 91.9, 91.9, 91.8, 91.7, 91.6, 91.6, 91.6, 91.5, 91.5, 91.4, 91.3, 91.1, 91.1, 91.0, 91.0, 91.0, 91.0, 90.8, 90.7, 90.7, 90.7, 90.6, 90.5, 90.2, 90.2, 90.1, 90.1, 90.0, 89.7, 89.7, 89.5, 89.5, 89.4, 89.0, 89.0, 88.9, 88.9, 88.7, 88.5, 88.5, 88.3, 88.3, 88.3, 88.1, 88.1, 87.9, 87.9, 87.8, 87.7, 87.6, 87.5, 87.5, 87.5, 87.5, 87.3, 87.0, 86.9, 86.8, 86.8, 86.7, 86.7, 86.6, 86.5, 86.4, 86.4, 86.4, 86.4, 86.4, 86.3, 86.3, 86.2, 86.2, 86.2, 86.1, 86.1, 86.1, 86.0, 85.9, 85.9, 85.9, 85.6, 85.4, 85.4, 85.3, 85.2, 85.2, 85.1, 85.0, 85.0, 84.9, 84.9, 84.7, 84.7, 84.6, 84.6, 84.4, 84.4, 84.3, 84.3, 84.3, 84.2, 84.1, 84.1, 84.1, 84.0, 83.9, 83.9, 83.9, 83.9, 83.8, 83.8, 83.6, 83.6, 83.2, 83.2, 83.1, 83.1, 83.1, 83.0, 83.0, 83.0, 82.9, 82.9, 82.9, 82.7, 82.7, 82.7, 82.5, 82.5, 82.5, 82.4, 82.1, 82.0, 82.0, 82.0, 82.0, 81.9, 81.8, 81.8, 81.6, 81.5, 81.5, 81.5, 81.4, 81.4, 81.2, 81.2, 81.2, 81.2, 81.1, 81.1, 81.1, 80.9, 80.8, 80.7, 80.7, 80.6, 80.5, 80.4, 80.4, 80.2, 80.1, 80.0, 80.0, 80.0, 79.9, 79.3, 79.3, 79.3, 79.3, 79.3, 79.0, 79.0, 79.0, 79.0, 78.8, 78.8, 78.6, 78.5, 78.4, 78.4, 78.3, 78.3, 78.2, 78.1, 78.1, 78.0, 77.8, 77.7, 77.6, 77.6, 77.4, 77.4, 77.4, 77.4, 77.3, 77.2, 77.2, 77.1, 77.1, 77.0, 77.0, 76.9, 76.9, 76.8, 76.7, 76.7, 76.6, 76.6, 76.6, 76.6, 76.6, 76.5, 76.5, 76.5, 76.2, 76.0, 76.0, 76.0, 76.0, 75.7, 75.5, 75.4, 75.2, 75.2, 75.1, 75.0, 75.0, 74.9, 74.8, 74.7, 74.7, 74.7, 74.3, 74.3, 74.2, 74.1, 74.1, 74.1, 74.0, 73.9, 73.9, 73.8, 73.7, 73.7, 73.6, 73.6, 73.5, 73.5, 73.5, 73.3, 73.3, 73.3, 73.2, 73.2, 73.2, 73.1, 73.1, 73.0, 73.0, 72.9, 72.9, 72.8, 72.8, 72.8, 72.8, 72.6, 72.5, 72.5, 72.5, 72.5, 72.4, 72.4, 72.4, 72.3, 72.3, 72.3, 72.2, 72.2, 72.0, 72.0, 72.0, 71.9, 71.9, 71.8, 71.8, 71.8, 71.7, 71.5, 71.4, 71.4, 71.3, 71.3, 71.1, 71.1, 71.0, 71.0, 70.9, 70.8, 70.7, 70.7, 70.7, 70.7, 70.6, 70.5, 70.5, 70.4, 70.4, 70.3, 70.3, 70.2, 70.2, 70.1, 70.0, 70.0, 69.9, 69.8, 69.6, 69.6, 69.5, 69.3, 69.2, 69.1, 69.1, 69.0, 69.0, 68.9, 68.8, 68.7, 68.7, 68.7, 68.5, 68.5, 68.2, 68.1, 68.0, 68.0, 68.0, 68.0, 67.9, 67.9, 67.8, 67.7, 67.6, 67.5, 67.5, 67.3, 67.3, 66.9, 66.9, 66.8, 66.8, 66.7, 66.6, 66.5, 66.4, 66.3, 66.2, 66.2, 66.1, 66.1, 66.0, 66.0, 65.9, 65.9, 65.9, 65.7, 65.7, 65.6, 65.6, 65.4, 65.3, 65.1, 65.1, 65.0, 65.0, 65.0, 64.9, 64.9, 64.8, 64.7, 64.7, 64.5, 64.5, 64.2, 64.2, 64.2, 64.0, 64.0, 63.8, 63.7, 63.7, 63.6, 63.6, 63.6, 63.6, 63.5, 63.4, 63.4, 63.3, 63.2, 63.0, 62.9, 62.9, 62.8, 62.8, 62.4, 62.3, 62.3, 62.3, 62.2, 62.1, 62.0, 62.0, 62.0, 62.0, 61.6, 61.5, 61.5, 61.5, 61.5, 61.5, 61.4, 61.4, 61.4, 61.2, 61.2, 61.1, 61.0, 61.0, 61.0, 60.9, 60.9, 60.9, 60.7, 60.6, 60.5, 60.5, 60.3, 60.2, 60.0, 60.0, 60.0, 60.0, 60.0, 59.8, 59.8, 59.6, 59.6, 59.4, 59.4, 59.2, 59.1, 59.0, 59.0, 58.5, 58.5, 58.5, 58.4, 58.1, 58.1, 58.1, 58.1, 58.0, 58.0, 58.0, 57.8, 57.8, 57.8, 57.8, 57.6, 57.5, 57.5, 57.5, 57.4, 57.3, 57.2, 57.2, 57.2, 57.2, 57.1, 57.1, 57.0, 57.0, 57.0, 57.0, 56.9, 56.8, 56.7, 56.6, 56.5, 56.5, 56.4, 56.4, 56.3, 56.3, 56.2, 56.1, 56.1, 56.0, 56.0, 55.7, 55.6, 55.5, 55.5, 55.5, 55.4, 55.4, 55.3, 55.3, 55.3, 55.0, 55.0, 55.0, 54.9, 54.8, 54.6, 54.6, 54.6, 54.5, 54.3, 54.2, 54.0, 53.9, 53.9, 53.5, 53.5, 53.5, 53.4, 53.3, 53.3, 53.2, 53.2, 53.1, 53.1, 52.9, 52.8, 52.8, 52.5, 52.5, 52.4, 52.4, 52.3, 52.3, 52.3, 52.2, 52.2, 52.2, 52.1, 52.0, 52.0, 51.9, 51.8, 51.8, 51.7, 51.6, 51.5, 51.5, 51.4, 51.4, 51.4, 51.3, 51.2, 51.2, 51.1, 51.1, 51.0, 51.0, 51.0, 50.9, 50.9, 50.8, 50.6, 50.5, 50.5, 50.3, 50.2, 50.1, 50.1, 50.0, 49.9, 49.7, 49.6, 49.4, 49.4, 49.1, 49.0, 49.0, 48.9, 48.9, 48.9, 48.9, 48.6, 48.5, 48.5, 48.4, 48.3, 48.3, 48.2, 48.0, 48.0, 48.0, 47.8, 47.8, 47.7, 47.6, 47.6, 47.5, 47.5, 47.2, 47.2, 47.2, 47.2, 47.0, 47.0, 47.0, 47.0, 47.0, 46.8, 46.8, 46.7, 46.5, 46.2, 46.2, 46.1, 46.1, 46.0, 46.0, 46.0, 46.0, 45.8, 45.7, 45.7, 45.6, 45.6, 45.5, 45.4, 45.4, 45.4, 45.2, 45.1, 45.0, 45.0, 45.0, 44.9, 44.9, 44.9, 44.9, 44.7, 44.5, 44.4, 44.3, 44.3, 44.3, 44.0, 43.9, 43.7, 43.6, 43.6, 43.6, 43.1, 42.5, 42.5, 42.5, 42.5, 42.5, 42.3, 41.8, 41.5, 41.5, 41.5, 41.4, 41.3, 41.2, 41.2, 41.0, 41.0, 41.0, 41.0, 40.9, 40.7, 40.7, 40.5, 40.4, 40.2, 40.1, 40.1, 40.0, 40.0, 39.8, 39.7, 39.6, 39.5, 39.3, 39.0, 39.0, 38.6, 38.5, 38.4, 38.3, 38.0, 38.0, 38.0, 37.5, 37.3, 37.2, 37.1, 37.1, 37.0, 37.0, 37.0, 37.0, 37.0, 36.8, 36.8, 36.7, 36.5, 36.5, 36.5, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 35.6, 35.5, 35.4, 35.1, 35.1, 35.0, 34.8, 34.8, 34.8, 34.5, 34.5, 34.3, 33.8, 33.8, 33.7, 33.5, 33.5, 33.2, 33.1, 32.7, 32.4, 32.0, 32.0, 31.6, 31.0, 31.0, 30.9, 30.9, 30.9, 30.5, 30.5, 30.2, 30.0, 30.0, 30.0, 29.8, 29.7, 29.4, 29.3, 29.0, 28.7, 28.5, 28.4, 28.1, 28.0, 28.0, 27.6, 27.5, 27.0, 27.0, 26.9, 26.8, 26.8, 26.7, 26.5, 26.4, 26.0, 26.0, 26.0, 25.7, 25.6, 25.0, 24.9, 24.1, 24.0, 23.6, 23.5, 23.5, 23.5, 23.4, 23.0, 23.0, 23.0, 23.0, 22.7, 22.5, 22.3, 22.0, 22.0, 21.4, 21.3, 21.0, 20.9, 20.9, 20.8, 20.5, 20.4, 20.2, 20.0, 19.5, 19.0, 18.3, 18.0, 18.0, 17.5, 17.4, 17.0, 16.9, 16.6, 16.4, 16.0, 16.0, 16.0, 16.0, 15.2, 15.0, 14.0, 13.7, 13.6, 13.0, 11.0, 11.0, 9.6, 9.0, 8.8, 7.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '.', 'Protein (g)']

After looking through every present data in Protein (g), we see a singular . in our data. This is likely what the 1,690th value is, which is more than likely a typo while working through the data.

Fat (g) will take care of that, along with removing the column names, due to it also being in our columns, making our number of data, 1,688.

First, we'll delete all the uneeded rows in our current dataframe, but we'll have to see what each column corresponds to. Looking through the excel sheet, I started counting every data point, along with dates from cel 478, and will be looking at it from index 470 to 480

In [8]:
df[470:480]
Out[8]:
Date Weight (kg) Waist (cm) BMI Unnamed: 10 Unnamed: 12 Unnamed: 14 Fast length (hours) Fast avg. 7 (hours) Height ... Fat (g) Carbohydrate % Protein % Fat % Macro score Score*Log. Kcal Weight (kg).1 Waist (cm).1 Unnamed: 40 Unnamed: 42
470 March 20th~ 49.9 58.7 16.481702 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
471 NaN 50.3 60.4 16.613820 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
472 NaN 50.0 59.3 16.514731 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
473 NaN 50.2 60.4 16.580790 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
474 NaN 49.8 59.6 16.448672 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
475 NaN 50.7 61.2 16.745937 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
476 NaN 50.3 59.7 16.613820 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
477 2020-03-30 00:00:00 50.8 61.7 16.778967 Kcals accumulated Daily intake (kcal) Kcals eaten (week) Kcal avg. week (day) Log. Kcal dangers Carbohydrate (g) ... Fat (g) Carbohydrate % Protein % Fat % Macro score Score*Log. Kcal Weight (kg) Waist (cm) Days/2 since tracking kcals Kcal avg. start
478 2020-03-30 00:00:00 50.2 59.7 16.580790 1424 1424 1424 1424 2.037701 206.79 ... 39.16 59.099743 15.718777 25.18148 81.217491 143.54043 50.2 59.7 1 NaN
479 2020-03-31 00:00:00 50.3 61.1 16.613820 1424 NaN 1424 1424 2.037701 NaN ... NaN NaN NaN NaN NaN NaN 50.3 61.1 2 1424

10 rows × 21 columns

List of columns to remove:¶

  • BMI - Essentially another column for weight, but in a different unit
  • Kcals accumulated + Kcals eaten (week) - Doesn't really provide extra data point for our prediction model. But I decided to keep Kcal avg. week (day), showing the average calories consumed for a day, from the previous 7 days.
  • Log. Kcal dangers - Similar to BMI, only ever used to warn myself of overeating, but have stopped using this metric.
  • Macro score - Calculates based on the macro % diet I've set, which is 30% Fat, 30% Protein, 40% Carbs. Lower the number, closer the macros are to target diet.
  • Score*Log Kcal - Uses Log. Kcal dangers multiplied with Macro score, another redundant feature that I don't use.
  • Weight (kg) + Waist (cm) - Don't need another weight and waist column with the same data.
  • Days/2 since tracking kcals - A form of indexing system for all the new data I've been recording and feature engineering.
  • Kcal avg. start - Measures mean caloric intake, by the number of results I've been recording.
In [9]:
# Remove mentioned rows above from data frame
df = df.drop(columns=['BMI', 'Unnamed: 10', 'Unnamed: 14', 'Fast avg. 7 (hours)', 'Macro score', 'Score*Log. Kcal', 'Weight (kg).1', 'Waist (cm).1', 'Unnamed: 40', 'Unnamed: 42'])
df[470:480]
Out[9]:
Date Weight (kg) Waist (cm) Unnamed: 12 Fast length (hours) Height Protein (g) Fat (g) Carbohydrate % Protein % Fat %
470 March 20th~ 49.9 58.7 NaN NaN NaN NaN NaN NaN NaN NaN
471 NaN 50.3 60.4 NaN NaN NaN NaN NaN NaN NaN NaN
472 NaN 50.0 59.3 NaN NaN NaN NaN NaN NaN NaN NaN
473 NaN 50.2 60.4 NaN NaN NaN NaN NaN NaN NaN NaN
474 NaN 49.8 59.6 NaN NaN NaN NaN NaN NaN NaN NaN
475 NaN 50.7 61.2 NaN NaN NaN NaN NaN NaN NaN NaN
476 NaN 50.3 59.7 NaN NaN NaN NaN NaN NaN NaN NaN
477 2020-03-30 00:00:00 50.8 61.7 Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
478 2020-03-30 00:00:00 50.2 59.7 1424 1424 206.79 55 39.16 59.099743 15.718777 25.18148
479 2020-03-31 00:00:00 50.3 61.1 NaN 1424 NaN NaN NaN NaN NaN NaN

We'll remove index 0 to 476, as it does not have the features we need to predict my weight or waist data.

In [10]:
df = df[477:]
df.head()
Out[10]:
Date Weight (kg) Waist (cm) Unnamed: 12 Fast length (hours) Height Protein (g) Fat (g) Carbohydrate % Protein % Fat %
477 2020-03-30 00:00:00 50.8 61.7 Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
478 2020-03-30 00:00:00 50.2 59.7 1424 1424 206.79 55 39.16 59.099743 15.718777 25.18148
479 2020-03-31 00:00:00 50.3 61.1 NaN 1424 NaN NaN NaN NaN NaN NaN
480 2020-03-31 00:00:00 50.0 59.9 1997 1710.5 283.65 38 74.8 57.893663 7.755893 34.350444
481 2020-04-01 00:00:00 50.8 61.0 NaN 1710.5 NaN NaN NaN NaN NaN NaN

Now lets change the title of the column to the one from index 477, then afterwards, removing index 477.

In [11]:
# Rename multiple columns
df = df.rename(columns={
    'Unnamed: 12':'Daily intake (kcal)',
    'Fast length (hours)':'Kcal avg. week (day)',
    'Height':'Carbohydrate (g)'
})

df=df[1:]
df
Out[11]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
478 2020-03-30 00:00:00 50.2 59.7 1424 1424 206.79 55 39.16 59.099743 15.718777 25.18148
479 2020-03-31 00:00:00 50.3 61.1 NaN 1424 NaN NaN NaN NaN NaN NaN
480 2020-03-31 00:00:00 50.0 59.9 1997 1710.5 283.65 38 74.8 57.893663 7.755893 34.350444
481 2020-04-01 00:00:00 50.8 61.0 NaN 1710.5 NaN NaN NaN NaN NaN NaN
482 2020-04-01 00:00:00 50.6 60.0 901 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528
... ... ... ... ... ... ... ... ... ... ... ...
30632 2066-07-15 00:00:00 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
30633 2066-07-16 00:00:00 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
30634 2066-07-16 00:00:00 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
30635 2066-07-17 00:00:00 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
30636 2066-07-17 00:00:00 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

30159 rows × 11 columns

Looking at Fat (g), you can see how data is entered every 2 cels, and on the date corner, there's a doubling on the sequential dates. So what I want to do, is remove any rows, that contain Nan in Fat (g), as only every 2 column of data is entered. In addition, the weight and waist has a morning and night check-ins, (hence the 2 repeating dates), but have moved towards only morning check-ins to save on time.

In [12]:
# Removing NaN rows, using 'Fat (g)' as our source of valid data.
df = df.dropna(subset=['Fat (g)'])

df
Out[12]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
478 2020-03-30 00:00:00 50.2 59.7 1424 1424 206.79 55 39.16 59.099743 15.718777 25.18148
480 2020-03-31 00:00:00 50.0 59.9 1997 1710.5 283.65 38 74.8 57.893663 7.755893 34.350444
482 2020-04-01 00:00:00 50.6 60.0 901 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528
484 2020-04-02 00:00:00 50.0 58.8 1578 1475 212.28 91 35.64 55.357655 23.73067 20.911675
486 2020-04-03 00:00:00 49.7 59.1 1870 1554 232.41 120 42.24 51.940999 26.818639 21.240362
... ... ... ... ... ... ... ... ... ... ... ...
3844 2024-11-07 00:00:00 73.4 74.0 2659 2911.571429 316.3 149.1 91.5 47.119288 22.211463 30.669249
3846 2024-11-08 00:00:00 74.2 75.0 3509 2967 263.5 198 179 30.488863 22.910038 46.601099
3848 2024-11-09 00:00:00 74.7 74.0 3732 3105.142857 302.3 229.5 175.5 32.622009 24.765964 42.612027
3850 2024-11-10 00:00:00 74.7 74.0 2812 3163.571429 247.6 162.3 121.3 36.261121 23.768901 39.969978
3852 2024-11-11 00:00:00 74.7 74.0 3086 3052.571429 295.1 184 118.9 39.524527 24.644232 35.831241

1688 rows × 11 columns

We now have 1688 valid data points to work with for this project, but we'll need to reset the index number for the dataframe.

In [13]:
# Reset the index of dataframe
df = df.reset_index(drop=True) # drop=True > drops our old index

df
Out[13]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
0 2020-03-30 00:00:00 50.2 59.7 1424 1424 206.79 55 39.16 59.099743 15.718777 25.18148
1 2020-03-31 00:00:00 50.0 59.9 1997 1710.5 283.65 38 74.8 57.893663 7.755893 34.350444
2 2020-04-01 00:00:00 50.6 60.0 901 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528
3 2020-04-02 00:00:00 50.0 58.8 1578 1475 212.28 91 35.64 55.357655 23.73067 20.911675
4 2020-04-03 00:00:00 49.7 59.1 1870 1554 232.41 120 42.24 51.940999 26.818639 21.240362
... ... ... ... ... ... ... ... ... ... ... ...
1683 2024-11-07 00:00:00 73.4 74.0 2659 2911.571429 316.3 149.1 91.5 47.119288 22.211463 30.669249
1684 2024-11-08 00:00:00 74.2 75.0 3509 2967 263.5 198 179 30.488863 22.910038 46.601099
1685 2024-11-09 00:00:00 74.7 74.0 3732 3105.142857 302.3 229.5 175.5 32.622009 24.765964 42.612027
1686 2024-11-10 00:00:00 74.7 74.0 2812 3163.571429 247.6 162.3 121.3 36.261121 23.768901 39.969978
1687 2024-11-11 00:00:00 74.7 74.0 3086 3052.571429 295.1 184 118.9 39.524527 24.644232 35.831241

1688 rows × 11 columns

In [14]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1688 entries, 0 to 1687
Data columns (total 11 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Date                  1688 non-null   object 
 1   Weight (kg)           1688 non-null   float64
 2   Waist (cm)            1688 non-null   float64
 3   Daily intake (kcal)   1688 non-null   object 
 4   Kcal avg. week (day)  1688 non-null   object 
 5   Carbohydrate (g)      1688 non-null   object 
 6   Protein (g)           1688 non-null   object 
 7   Fat (g)               1688 non-null   object 
 8   Carbohydrate %        1688 non-null   object 
 9   Protein %             1688 non-null   object 
 10  Fat %                 1688 non-null   object 
dtypes: float64(2), object(9)
memory usage: 145.2+ KB

Despite all columns except being floats, most of then are objects. Let's convert them into float columns.

In [15]:
# Convert column object into float
convert = ['Daily intake (kcal)', 'Kcal avg. week (day)', 'Carbohydrate (g)', 'Protein (g)', 'Fat (g)', 'Carbohydrate %', 'Protein %', 'Fat %'] # all columns that need to be converted
df[convert] = df[convert].astype(float)

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1688 entries, 0 to 1687
Data columns (total 11 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Date                  1688 non-null   object 
 1   Weight (kg)           1688 non-null   float64
 2   Waist (cm)            1688 non-null   float64
 3   Daily intake (kcal)   1688 non-null   float64
 4   Kcal avg. week (day)  1688 non-null   float64
 5   Carbohydrate (g)      1688 non-null   float64
 6   Protein (g)           1688 non-null   float64
 7   Fat (g)               1688 non-null   float64
 8   Carbohydrate %        1688 non-null   float64
 9   Protein %             1688 non-null   float64
 10  Fat %                 1688 non-null   float64
dtypes: float64(10), object(1)
memory usage: 145.2+ KB

Data interpretation/visualization¶

Now most of the data has been cleaned, we can make some initial interpretations of the data, and visually see them as graphs.

In [16]:
df.describe()
Out[16]:
Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
count 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000
mean 60.912915 67.264502 2480.717417 2477.837438 258.473904 106.511671 111.140604 42.339985 16.806491 40.567946
std 6.187260 4.024980 840.246992 744.614903 112.001458 60.502474 45.456362 13.190117 6.866947 11.365782
min 48.600000 58.200000 0.000000 1102.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
25% 58.600000 64.600000 1991.000000 2006.357143 184.550000 61.200000 82.100000 34.665653 11.893795 34.012749
50% 59.600000 66.350000 2187.000000 2199.071429 245.700000 91.650000 105.600000 43.072585 16.402331 40.153975
75% 62.300000 70.300000 3001.500000 2890.357143 316.925000 146.800000 138.100000 50.688393 20.810325 47.398633
max 76.700000 77.900000 5231.000000 4453.428571 696.600000 374.400000 318.600000 100.000000 57.626769 78.334271

Weight vs Waist: The numbers are reasonably similar to each other, which could mean a change in one, is most likely to change the other as well.

Daily intake (kcal): We can see there are data points with no calories consumed. Because I knew the data I put in, it's days where I did extended fasts, and ate nothing within that day.

Carbohydrate %: Max is 100%, which is unusual to only be consuming carbs and no protein and fats. Again, since I knew the data, it was a day when I only drank a bottle of ginger beer. Soft drink with only sugars.

Let's create graphs and visualize the overall change in weight overtime

In [17]:
# Create weight to date graph
fig, ax = plt.subplots(figsize=(20,10)) # set figure size, and create two variables for the figure, and axis within the figure

ax.set(ylabel='Weight (kg)', # set label names for x and y
       xlabel='Date')

ax.plot(df['Date'].astype(str), df['Weight (kg)']); # plotting Date (but also converting it into string from object), and Weight
No description has been provided for this image

Uh oh, looks like we did the date column wrong, as it was a object, and we simply only converted it as string. Lets change it to datetime value and see if it fixes things.

In [18]:
df['Date'] = pd.to_datetime(df['Date'])

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1688 entries, 0 to 1687
Data columns (total 11 columns):
 #   Column                Non-Null Count  Dtype         
---  ------                --------------  -----         
 0   Date                  1688 non-null   datetime64[ns]
 1   Weight (kg)           1688 non-null   float64       
 2   Waist (cm)            1688 non-null   float64       
 3   Daily intake (kcal)   1688 non-null   float64       
 4   Kcal avg. week (day)  1688 non-null   float64       
 5   Carbohydrate (g)      1688 non-null   float64       
 6   Protein (g)           1688 non-null   float64       
 7   Fat (g)               1688 non-null   float64       
 8   Carbohydrate %        1688 non-null   float64       
 9   Protein %             1688 non-null   float64       
 10  Fat %                 1688 non-null   float64       
dtypes: datetime64[ns](1), float64(10)
memory usage: 145.2 KB
In [19]:
fig, ax = plt.subplots(figsize=(20,10))

ax.set(ylabel='Weight (kg)',
       xlabel='Date')

ax.plot(df['Date'], df['Weight (kg)']);
No description has been provided for this image

We can see the trends of weight loss, weight gain, and weight maintaining. Let's try and compare it to other data points.

First we'll try waist measurement, and see if our first hypothesis of both being somwhat correlated checks out.

In [20]:
fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement overtime', xlabel='Date')

ax.plot(df['Date'], df['Weight (kg)'], label='Weight in kg')
ax.plot(df['Date'], df['Waist (cm)'], label='Waist in cm')
ax.legend();
No description has been provided for this image

It is clear that if I gained on either one of the 2 variables, then the the other variable will also increase as well.

I also see how the waist measurement numbers, starts to meet up closely with the weight measurement numbers. I've been consistently going to the gym since Q3 of 2023, and it could mean that building up muscle or lowering body fat has the consequence of minimizing waist circumference. Though I won't truly know the conclusion, as I don't consistently track my body fat %, and neither are the tools I have that accurate to track such data point.

Let's also compare weight with our other features, that aim to predict weight + waist size.

In [301]:
fig, ax1 = plt.subplots(nrows=2, # create 4 different graphs, comparing 4 different columns to the weight data
                       ncols=2, # ax1 > adding a 1 to it, allows us to create a secondary y axis, due to varying values between 2 different data
                       figsize=(20,10))

ax1[0,0].set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1[0,0].plot(df['Date'], df['Daily intake (kcal)'], label='Calorie consumption in kcal', color='orange', linewidth=0.5) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1[0,0].twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[0,0].legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()

# rinse and repeat
ax1[0,1].set(title='Weight and Carbohydrate % consumption overtime', xlabel='Date')
ax1[0,1].plot(df['Date'], df['Carbohydrate %'], label='Carbohydrate consumption %', color='orange', linewidth=0.5)
ax3 = ax1[0,1].twinx()
ax3.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[0,1].legend(loc='lower right')
ax3.legend()

ax1[1,0].set(title='Weight and Protein % consumption overtime', xlabel='Date')
ax1[1,0].plot(df['Date'], df['Protein %'], label='Protein consumption %', color='orange', linewidth=0.5)
ax4 = ax1[1,0].twinx()
ax4.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[1,0].legend(loc='lower center')
ax4.legend()

ax1[1,1].set(title='Weight and Fat % consumption overtime', xlabel='Date')
ax1[1,1].plot(df['Date'], df['Fat %'], label='Fat consumption %', color='orange', linewidth=0.5)
ax5 = ax1[1,1].twinx()
ax5.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[1,1].legend(loc='lower center')
ax5.legend();
No description has been provided for this image

There's clear correlation between higher intake of calories, leading to more weight gain. And then vice versa with lower caloric intake.

Though macro % consumption is a bit less conclusive than calories. Take carbohydrate %, there's higher percentage in consumption when I was ~50kg and a somwhat decrease within the weight gain in early 2021, which might lead to thinking that higher carbs lead to lower weight. But carbohydrate % consumption stays quite leveled throughout the rest of the chart, and shows varying levels of weight gain and loss, making it quite inconclusive.

Fat % is a bit more inconclusive. We see a rise in fat % consumption, with the rise in weight in early 2021. Then it gradually goes down and flattens itself out for the rest of the graph, seemingly with zero to no correlation to future weight gain or loss.

Maybe protein % has a miniscule change in the weight loss, as the consumption stays somwhat stable, but there's a slight more consumption in protein, where weight has slowly been lost, at the end of the graph.

Graph Anomaly:¶

There is an anomaly in the caloric/weight graph. Near Q3 of 2021, there's an uptake in caloric consumption, yet weight had remained stable or had went down. This was a period of time where I took the chemical, dnp, known for its weight loss effect, and often used by bodybuilders; and must likelly be the cause of the graph anomaly.

Let's look at Q3 and Q4 of 2021, and have a better look at it

In [298]:
# plot Q3 and Q4 2021 (july till december) to better look at anomaly in caloric vs weight graph
anomaly_df = (df['Date'] >= '2021-07-01') & (df['Date'] <= '2021-12-31') # returns True and False, matching to the boolean operation we've given. Is it between july and december?
anomaly = df[anomaly_df] # putting boolean values into dataframe, to than only give us a dataframe, containing True

fig, ax1 = plt.subplots(figsize=(20,5))

ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(anomaly['Date'], anomaly['Daily intake (kcal)'], label='Calorie consumption in kcal', color='orange', linewidth=2) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(anomaly['Date'], anomaly['Weight (kg)'], label='Weight in kg', linewidth=2)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend();
No description has been provided for this image

Now we get a better visual of the graph anomaly. Where weight was quite consistent on the days that caloric consumption was ~2000kcal, but notice how weight was loss, despite higher consumption of calories, reaching >3000kcal for the September and October period. Looking back at the original excel sheet, there's a stated dnp dosage column, showing evidence and the amount of dnp that was consumed, between the periods of September and October.

We'll have to remove Sep-2021 and Oct-2021 from our data, to prevent third-party influence from changing our data. This does mean that there'd be a very big jump in weight difference from our previous weight data (08/31) to the next data (11/01), but is better than having anomalies in the data.

In [22]:
# Remove 2021/09 and 2021/10 due to anomaly data that is likely affected from third-party influence
anomaly_date = ~((df['Date'] >= '2021-09-01') & (df['Date'] <= '2021-10-31')) # find 2021/09 and 2021/10, with `not` symbol applied (~)
clean_df = df[anomaly_date].reset_index(drop=True)
clean_df
Out[22]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat %
0 2020-03-30 50.2 59.7 1424.0 1424.000000 206.79 55.0 39.16 59.099743 15.718777 25.181480
1 2020-03-31 50.0 59.9 1997.0 1710.500000 283.65 38.0 74.80 57.893663 7.755893 34.350444
2 2020-04-01 50.6 60.0 901.0 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528
3 2020-04-02 50.0 58.8 1578.0 1475.000000 212.28 91.0 35.64 55.357655 23.730670 20.911675
4 2020-04-03 49.7 59.1 1870.0 1554.000000 232.41 120.0 42.24 51.940999 26.818639 21.240362
... ... ... ... ... ... ... ... ... ... ... ...
1622 2024-11-07 73.4 74.0 2659.0 2911.571429 316.30 149.1 91.50 47.119288 22.211463 30.669249
1623 2024-11-08 74.2 75.0 3509.0 2967.000000 263.50 198.0 179.00 30.488863 22.910038 46.601099
1624 2024-11-09 74.7 74.0 3732.0 3105.142857 302.30 229.5 175.50 32.622009 24.765964 42.612027
1625 2024-11-10 74.7 74.0 2812.0 3163.571429 247.60 162.3 121.30 36.261121 23.768901 39.969978
1626 2024-11-11 74.7 74.0 3086.0 3052.571429 295.10 184.0 118.90 39.524527 24.644232 35.831241

1627 rows × 11 columns

Feature engineering¶

After seeing how different variables affect weight gain specifically, and that waist size also has heavy correlation to weight as well, there's a big problem with predicting weight and waist. It's not a typical regression problem, where you predict a value of something, but we're predicting the loss and gain of 2 different variables, based on other datapoints.

It's prediction will then be added onto the current weight and waist values, and will continue it's prediction with its previously predicted value. This does make ML accuracy even harder, as we're also relying on our predicted value to then make another prediction.

Anomaly issue¶

Because we got a portion of data removed, there's a substantial difference in weight and possibly waist size between the data points, outside the anomaly time frame. So we'll do our feature engineering on the dataframe, before date anomaly removal, to get the data point difference of its actual neighbour.

In [23]:
# Create features on weight and waist gain/loss
df['Weight Difference'] = df['Weight (kg)'].diff()
df['Waist Difference'] = df['Waist (cm)'].diff()

# Display the DataFrame with the new columns
print(df)

           Date  Weight (kg)  Waist (cm)  Daily intake (kcal)  \
0    2020-03-30         50.2        59.7               1424.0   
1    2020-03-31         50.0        59.9               1997.0   
2    2020-04-01         50.6        60.0                901.0   
3    2020-04-02         50.0        58.8               1578.0   
4    2020-04-03         49.7        59.1               1870.0   
...         ...          ...         ...                  ...   
1683 2024-11-07         73.4        74.0               2659.0   
1684 2024-11-08         74.2        75.0               3509.0   
1685 2024-11-09         74.7        74.0               3732.0   
1686 2024-11-10         74.7        74.0               2812.0   
1687 2024-11-11         74.7        74.0               3086.0   

      Kcal avg. week (day)  Carbohydrate (g)  Protein (g)  Fat (g)  \
0              1424.000000            206.79         55.0    39.16   
1              1710.500000            283.65         38.0    74.80   
2              1440.666667             82.35         61.5    29.92   
3              1475.000000            212.28         91.0    35.64   
4              1554.000000            232.41        120.0    42.24   
...                    ...               ...          ...      ...   
1683           2911.571429            316.30        149.1    91.50   
1684           2967.000000            263.50        198.0   179.00   
1685           3105.142857            302.30        229.5   175.50   
1686           3163.571429            247.60        162.3   121.30   
1687           3052.571429            295.10        184.0   118.90   

      Carbohydrate %  Protein %      Fat %  Weight Difference  \
0          59.099743  15.718777  25.181480                NaN   
1          57.893663   7.755893  34.350444               -0.2   
2          38.997017  29.123455  31.879528                0.6   
3          55.357655  23.730670  20.911675               -0.6   
4          51.940999  26.818639  21.240362               -0.3   
...              ...        ...        ...                ...   
1683       47.119288  22.211463  30.669249               -0.4   
1684       30.488863  22.910038  46.601099                0.8   
1685       32.622009  24.765964  42.612027                0.5   
1686       36.261121  23.768901  39.969978                0.0   
1687       39.524527  24.644232  35.831241                0.0   

      Waist Difference  
0                  NaN  
1                  0.2  
2                  0.1  
3                 -1.2  
4                  0.3  
...                ...  
1683               0.1  
1684               1.0  
1685              -1.0  
1686               0.0  
1687               0.0  

[1688 rows x 13 columns]
In [ ]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1688 entries, 0 to 1687
Data columns (total 13 columns):
 #   Column                Non-Null Count  Dtype         
---  ------                --------------  -----         
 0   Date                  1688 non-null   datetime64[ns]
 1   Weight (kg)           1688 non-null   float64       
 2   Waist (cm)            1688 non-null   float64       
 3   Daily intake (kcal)   1688 non-null   float64       
 4   Kcal avg. week (day)  1688 non-null   float64       
 5   Carbohydrate (g)      1688 non-null   float64       
 6   Protein (g)           1688 non-null   float64       
 7   Fat (g)               1688 non-null   float64       
 8   Carbohydrate %        1688 non-null   float64       
 9   Protein %             1688 non-null   float64       
 10  Fat %                 1688 non-null   float64       
 11  Weight Difference     1687 non-null   float64       
 12  Waist Difference      1687 non-null   float64       
dtypes: datetime64[ns](1), float64(12)
memory usage: 171.6 KB
In [ ]:
# index 0 has NaN, due to no data point preceding it. We'll change it to = 0
df = df.fillna(0)
In [ ]:
df
Out[ ]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat % Weight Difference Waist Difference
0 2020-03-30 50.2 59.7 1424.0 1424.000000 206.79 55.0 39.16 59.099743 15.718777 25.181480 0.0 0.0
1 2020-03-31 50.0 59.9 1997.0 1710.500000 283.65 38.0 74.80 57.893663 7.755893 34.350444 -0.2 0.2
2 2020-04-01 50.6 60.0 901.0 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528 0.6 0.1
3 2020-04-02 50.0 58.8 1578.0 1475.000000 212.28 91.0 35.64 55.357655 23.730670 20.911675 -0.6 -1.2
4 2020-04-03 49.7 59.1 1870.0 1554.000000 232.41 120.0 42.24 51.940999 26.818639 21.240362 -0.3 0.3
... ... ... ... ... ... ... ... ... ... ... ... ... ...
1683 2024-11-07 73.4 74.0 2659.0 2911.571429 316.30 149.1 91.50 47.119288 22.211463 30.669249 -0.4 0.1
1684 2024-11-08 74.2 75.0 3509.0 2967.000000 263.50 198.0 179.00 30.488863 22.910038 46.601099 0.8 1.0
1685 2024-11-09 74.7 74.0 3732.0 3105.142857 302.30 229.5 175.50 32.622009 24.765964 42.612027 0.5 -1.0
1686 2024-11-10 74.7 74.0 2812.0 3163.571429 247.60 162.3 121.30 36.261121 23.768901 39.969978 0.0 0.0
1687 2024-11-11 74.7 74.0 3086.0 3052.571429 295.10 184.0 118.90 39.524527 24.644232 35.831241 0.0 0.0

1688 rows × 13 columns

In [297]:
df.describe()
Out[297]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat % Weight Difference Waist Difference
count 1688 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1688.000000 1687.000000 1687.000000
mean 2022-07-21 11:59:59.999999744 60.912915 67.264502 2480.717417 2477.837438 258.473904 106.511671 111.140604 42.339985 16.806491 40.567946 0.014523 0.008477
min 2020-03-30 00:00:00 48.600000 58.200000 0.000000 1102.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 -2.500000 -5.600000
25% 2021-05-25 18:00:00 58.600000 64.600000 1991.000000 2006.357143 184.550000 61.200000 82.100000 34.665653 11.893795 34.012749 -0.300000 -0.400000
50% 2022-07-21 12:00:00 59.600000 66.350000 2187.000000 2199.071429 245.700000 91.650000 105.600000 43.072585 16.402331 40.153975 0.000000 0.000000
75% 2023-09-16 06:00:00 62.300000 70.300000 3001.500000 2890.357143 316.925000 146.800000 138.100000 50.688393 20.810325 47.398633 0.300000 0.500000
max 2024-11-11 00:00:00 76.700000 77.900000 5231.000000 4453.428571 696.600000 374.400000 318.600000 100.000000 57.626769 78.334271 1.500000 4.700000
std NaN 6.187260 4.024980 840.246992 744.614903 112.001458 60.502474 45.456362 13.190117 6.866947 11.365782 0.469796 0.824128
In [24]:
# Now remove the anomaly dates
anomaly_date = ~((df['Date'] >= '2021-09-01') & (df['Date'] <= '2021-10-31'))
clean_df = df[anomaly_date].reset_index(drop=True)
clean_df
Out[24]:
Date Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat % Weight Difference Waist Difference
0 2020-03-30 50.2 59.7 1424.0 1424.000000 206.79 55.0 39.16 59.099743 15.718777 25.181480 NaN NaN
1 2020-03-31 50.0 59.9 1997.0 1710.500000 283.65 38.0 74.80 57.893663 7.755893 34.350444 -0.2 0.2
2 2020-04-01 50.6 60.0 901.0 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528 0.6 0.1
3 2020-04-02 50.0 58.8 1578.0 1475.000000 212.28 91.0 35.64 55.357655 23.730670 20.911675 -0.6 -1.2
4 2020-04-03 49.7 59.1 1870.0 1554.000000 232.41 120.0 42.24 51.940999 26.818639 21.240362 -0.3 0.3
... ... ... ... ... ... ... ... ... ... ... ... ... ...
1622 2024-11-07 73.4 74.0 2659.0 2911.571429 316.30 149.1 91.50 47.119288 22.211463 30.669249 -0.4 0.1
1623 2024-11-08 74.2 75.0 3509.0 2967.000000 263.50 198.0 179.00 30.488863 22.910038 46.601099 0.8 1.0
1624 2024-11-09 74.7 74.0 3732.0 3105.142857 302.30 229.5 175.50 32.622009 24.765964 42.612027 0.5 -1.0
1625 2024-11-10 74.7 74.0 2812.0 3163.571429 247.60 162.3 121.30 36.261121 23.768901 39.969978 0.0 0.0
1626 2024-11-11 74.7 74.0 3086.0 3052.571429 295.10 184.0 118.90 39.524527 24.644232 35.831241 0.0 0.0

1627 rows × 13 columns

In [25]:
fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement overtime', xlabel='Date')

ax.plot(clean_df['Date'], clean_df['Weight Difference'], label='Weight in kg')
ax.plot(clean_df['Date'], clean_df['Waist Difference'], label='Waist in cm')
ax.legend();
No description has been provided for this image

Evaluation change¶

Will be changing the prediction of weight and waist size, to the weight and waist difference, as the caloric and macro intake, in addition to the current weight/waist measurement, will have an affect on the change in weight and waist. This project will create to ML models, with one predicting weight difference, and the other for waist difference.

Add datetime parameters for dataframe¶

ML models cannot use datetime values to predict things. So to boost up the number of features that could possibly help with prediction, we will split them up into integers that the ML models can read/use.

In [26]:
# Create separated datetime variables, as integers for ML models to read
ml_df = clean_df.copy() # changing dataframe name to something shorter to rewrite constantly + dulpicating it with .copy()
ml_df['Year'] = ml_df.Date.dt.year
ml_df['Month'] = ml_df.Date.dt.month
ml_df['Day'] = ml_df.Date.dt.day
ml_df['Dayofweek'] = ml_df.Date.dt.dayofweek
ml_df['Dayofyear'] = ml_df.Date.dt.dayofyear

# Drop the date column
ml_df.drop('Date', axis=1, inplace=True) # axis=1 > stating that we're searching by column, not row (0). inplace=True > modifies current dataframe in memory, applying changes while running code
ml_df
Out[26]:
Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat % Weight Difference Waist Difference Year Month Day Dayofweek Dayofyear
0 50.2 59.7 1424.0 1424.000000 206.79 55.0 39.16 59.099743 15.718777 25.181480 NaN NaN 2020 3 30 0 90
1 50.0 59.9 1997.0 1710.500000 283.65 38.0 74.80 57.893663 7.755893 34.350444 -0.2 0.2 2020 3 31 1 91
2 50.6 60.0 901.0 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528 0.6 0.1 2020 4 1 2 92
3 50.0 58.8 1578.0 1475.000000 212.28 91.0 35.64 55.357655 23.730670 20.911675 -0.6 -1.2 2020 4 2 3 93
4 49.7 59.1 1870.0 1554.000000 232.41 120.0 42.24 51.940999 26.818639 21.240362 -0.3 0.3 2020 4 3 4 94
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
1622 73.4 74.0 2659.0 2911.571429 316.30 149.1 91.50 47.119288 22.211463 30.669249 -0.4 0.1 2024 11 7 3 312
1623 74.2 75.0 3509.0 2967.000000 263.50 198.0 179.00 30.488863 22.910038 46.601099 0.8 1.0 2024 11 8 4 313
1624 74.7 74.0 3732.0 3105.142857 302.30 229.5 175.50 32.622009 24.765964 42.612027 0.5 -1.0 2024 11 9 5 314
1625 74.7 74.0 2812.0 3163.571429 247.60 162.3 121.30 36.261121 23.768901 39.969978 0.0 0.0 2024 11 10 6 315
1626 74.7 74.0 3086.0 3052.571429 295.10 184.0 118.90 39.524527 24.644232 35.831241 0.0 0.0 2024 11 11 0 316

1627 rows × 17 columns

DATA LEAK ISSUE!!!¶

I've just picked up on an issue between Weight Difference and Weight (kg). The difference is stated from the previous day to the present day. Aka from 50.2kg, to 50.0kg, the weight difference is -0.2kg. This becomes a big problem, as this becomes a loophole for the ML model to get the answer keys. By simply finding the difference of the present and previous days on weight, and then putting that as their prediction, they get the answer. Defying the project's mission on weight and waist prediction!

A similar thing happens with Waist Difference too. There's huge correlation between weight loss and waist loss. Having Waist Difference in the data, makes the ML model too reliant on this specific datapoint to guess Weight Difference, and again, defeats the purpose of the project with weight and waist prediction.

In addition, the end goal is to predict both values of weight and waist. If that's the case, then there is zero possibility that you'd have access to either data on the difference between present and future, as the ML model doesn't know the future!

Solution¶

Fortunately there's a work around, where I could still include Weight (kg) in y_train. By moving the Weight Difference rows by -1 index. What this does, is instead of showing the difference from previous, and present day, it shows the difference between present, and next day! We've got all the data point for the ML model to make a prediction on what my change in weight is going to be on the next day, and avoids the data leakage where the ML model has the ability to cheat the answer from a feature in y_train.

We will also remove the 'Difference' volume, that is not used in X_train, but sits in y_train, as it has a very likely chance of being used by the ML model, to predict the weight difference.

In [27]:
# shift both weight and waist difference by -1 index
ml_df['Weight Difference'] = ml_df['Weight Difference'].shift(-1)
ml_df['Waist Difference'] = ml_df['Waist Difference'].shift(-1)

# fill nan at end of index with 0
ml_df = ml_df.fillna(0)
ml_df
Out[27]:
Weight (kg) Waist (cm) Daily intake (kcal) Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) Carbohydrate % Protein % Fat % Weight Difference Waist Difference Year Month Day Dayofweek Dayofyear
0 50.2 59.7 1424.0 1424.000000 206.79 55.0 39.16 59.099743 15.718777 25.181480 -0.2 0.2 2020 3 30 0 90
1 50.0 59.9 1997.0 1710.500000 283.65 38.0 74.80 57.893663 7.755893 34.350444 0.6 0.1 2020 3 31 1 91
2 50.6 60.0 901.0 1440.666667 82.35 61.5 29.92 38.997017 29.123455 31.879528 -0.6 -1.2 2020 4 1 2 92
3 50.0 58.8 1578.0 1475.000000 212.28 91.0 35.64 55.357655 23.730670 20.911675 -0.3 0.3 2020 4 2 3 93
4 49.7 59.1 1870.0 1554.000000 232.41 120.0 42.24 51.940999 26.818639 21.240362 0.4 1.2 2020 4 3 4 94
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
1622 73.4 74.0 2659.0 2911.571429 316.30 149.1 91.50 47.119288 22.211463 30.669249 0.8 1.0 2024 11 7 3 312
1623 74.2 75.0 3509.0 2967.000000 263.50 198.0 179.00 30.488863 22.910038 46.601099 0.5 -1.0 2024 11 8 4 313
1624 74.7 74.0 3732.0 3105.142857 302.30 229.5 175.50 32.622009 24.765964 42.612027 0.0 0.0 2024 11 9 5 314
1625 74.7 74.0 2812.0 3163.571429 247.60 162.3 121.30 36.261121 23.768901 39.969978 0.0 0.0 2024 11 10 6 315
1626 74.7 74.0 3086.0 3052.571429 295.10 184.0 118.90 39.524527 24.644232 35.831241 0.0 0.0 2024 11 11 0 316

1627 rows × 17 columns

Train test split¶

Let's split the dataset into a train test split. I feel like there's not a whole lot of datapoint I've collected, so this dataset won't include a validation dataset, and the test will be from May 2024 onwards. The point where weight/waist gain start to become a loss.

In [28]:
# Splitting data into test and train datasets
df_test = ml_df[(ml_df.Year == 2024) & (ml_df.Month > 5)]
df_train = ml_df[~((ml_df.Year == 2024) & (ml_df.Month > 5))]

len(df_test), len(df_train)
Out[28]:
(164, 1463)
In [29]:
164/1463
Out[29]:
0.11209842788790157

We will be using about 11% of the data for testing

Now we'll have to split it into x, and y dataset. And into 2 separate x and y's as well for the weight and waist.

We will also remove the corresponding difference column, that is not used by x to prevent unfair advantage to the model.

In addition, I am forecasting my weight, and I realistically wouldn't have a forecast of my waist size on the following day. I only have my current weight, waist, and food consumption from the current day.

In [117]:
# Splitting dataframe into weight and waist, to train 2 ML models
Xweight_train, yweight_train = df_train.drop(['Weight Difference', 'Waist Difference'], axis=1), df_train['Weight Difference']
Xwaist_train, ywaist_train = df_train.drop(['Weight Difference', 'Waist Difference'], axis=1), df_train['Waist Difference']

Xweight_test, yweight_test = df_test.drop(['Weight Difference', 'Waist Difference'], axis=1), df_test['Weight Difference']
Xwaist_test, ywaist_test = df_test.drop(['Weight Difference', 'Waist Difference'], axis=1), df_test['Waist Difference']

# Check the shapes
Xweight_train.shape, yweight_train.shape, Xweight_test.shape, yweight_test.shape
Out[117]:
((1463, 15), (1463,), (164, 15), (164,))

Building evaluation metrics¶

To gauge on accuracy, I'm focusing on the percentage absolute difference between prediction and truth. I don't think some of the metrics work well like MAE (mean absolute error), as the values are usually in the decimals, rather than in whole numbers. Though we will include r^2 and MSE (mean squared error), as to identify the larger errors, since our value variation is so small.

sklearn doesn't have a metric for MAPE (mean absolute percentage error), so we will form one from numpy.

In [55]:
from sklearn.metrics import r2_score

# Create MAPE evaluation metric from numpy
def mean_absolute_percent_error(y_true, y_pred):
    y_true = np.ravel(y_true)  # Ensures 1D shape
    y_pred = np.ravel(y_pred)

    y_true, y_pred = np.array(y_true), np.array(y_pred)
    dif = np.abs(y_true - y_pred)

    percent_error = np.zeros_like(y_true, dtype=float)

    # Task1 > If true is 0, and pred is not 0, set to 100%
    percent_error[(y_true == 0) & (y_pred != 0)] = 100

    # Task2 > If true and pred is 0, set to 0%
    percent_error[(y_true == 0) & (y_pred == 0)] = 0

    # Task3 > All other cases are then calculated
    mask = (y_true != 0) # boolean array of where y_true is 0 or not
    percent_error[mask] = (np.abs(y_true[mask] - y_pred[mask]) / np.abs(y_true[mask])) * 100 # absolute dif of true and pred, divided by true. Multiply bvy 100 to get percentage

    return np.mean(percent_error)

# also add in mse. r^2 is built in with model scoring
from sklearn.metrics import mean_squared_error

# create function to include all evaluation metric in one space when testing models
def show_scores(model, truth, test, title):
    
    # model predicting values from training
    preds = model.predict(test)

    truth = np.squeeze(truth)
    preds = np.squeeze(preds)

    # calculating scores
    scores = {
        "Test MAPE (%)":mean_absolute_percent_error(truth, preds),
        "Test MSE":mean_squared_error(truth, preds),
        "Test R^2":r2_score(truth, preds) # model scoring for r^2, done by doing prediction to X, to y
    }

    return title, scores

Tweaking absolute percent error¶

I thought it'd be easy to form this evaluation metric in a function, but because we're dealing with division, 0's can't be used or I'll cause an error for Infinity. It doesn't help that I'm dealing with very low values, that often hit 0 sometimes. So there's a few tasks to run, in order to avoid all possible forms of errors.

Finding the difference of true, and prediction, then dividing by value of true is the main core of the evaluation. Firstly, we'll have to remove all 0's from True as it'd cause an error. If True and Prediction is 0, then it has no difference, making 0%. Now with True being 0 and Pred being a value, it'll cause infinity. What we'll do is default that to 100%.

It does make it slightly unfair, as for example, True is 0.1, and Prediction is 2, that makes 2000% difference, but is much better than a non-comprehensible value that we can't interpret from.

Algorithm choice¶

We're going to have a look at what is the right algorithm for our regression problem. Python has this cheatsheet to help us discern the right models to use.

  • YES - >50 samples
  • NO - predicting a category
  • YES - predicting a quantity
  • YES - <100k samples
  • YES - few features should be important

This leaves us with two algorithms, Lasso and ElasticNet, and I will be trying both algorithms to see which performs best overall.

Lasso:¶

Known as Least Absolute Shrinkage and Selection Operator.

Least Absolute means the usage of L1 regularization technique, where we look at the sum of the absolute values (difference) of the coefficients (relationship between the predicting variables, vs the target variable). Shrinkage means shrinking the unimportant features towards zero, in terms of the penalty of L1, in order to reduce complexity of the model, and prevent overfitting.

Selection means selecting only the most significant predictors to the target variable, and excluding less important features in the model, in order to unhance model interpretability and performance. Operator means applying the mathematical operations to L1 penalties to the coefficients during model training.

TLDR: Lasso model both shrinks, by reducing the ability to overfit, and selects, by simplifying the model and excluding irrelevant features¶

ElasticNet:¶

Combination of Lasso Regression and Ridge Regression.

Lasso Regression (L1 Regularization) looks at the penalties, proportional to the difference of value to the coefficients (relationship between prediciting, and targeting variables). Some coefficients reach 0, meaning it has effectively selected this feature. It's chosen due to its likelihood to intersect with the most optimal solution.

Ridge Regression (L2 Regularization) means squaring the coefficients (relationship between the predicting variable, vs the target variable) as its penalty term. It penalizes larger coefficients more heavily due to squaring, but is the opposite with smaller coefficients, in order to prevent model from relying too heavily on any singular feature.

TLDR: Uses L1 and L2 Regularization, striking a balance betwee Ridge and Lasso regression¶

Model Training: Lasso¶

We'll try out the model on Lasso first

In [ ]:
from sklearn.linear_model import Lasso

model_weight = Lasso()
model_weight.fit(Xweight_train, yweight_train)

model_waist = Lasso()
model_waist.fit(Xwaist_train, ywaist_train)
show_scores(model_weight, yweight_test, Xweight_test,'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(98.06115173377636),
   'Test MSE': 0.35490734435726756,
   'Test R^2': 0.03119197823253428}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(101.3501738153358),
   'Test MSE': 0.6202001018438599,
   'Test R^2': -0.002728014464978612}))

Results¶

MSE and R^2 have very miniscule values. Despite looking really good in terms of evaluatoin value from your typical regression problem, we can't really tell how accurate the predictions really are. However MAPE has provided some good insight over the other two, as we can see its roughly 99.9% different from the truth value. Saying if True is 2, then Prediction could be 4 or 0.

Based on that, the prediction is not really the best, and feels a bit odd that the ML model is predicting at 100% difference from the truth. Could it be defaulting to 0, just because the % difference would likely be lower than whichever prediction they make?

Data scaling¶

One thing suggested with model training, is normalizing/scaling the data. If variable vary way too much from each other, it can cause a bit of confusion with the model training on it. What we do is replace it based on standard deviation, allowing very constant value scaling, and proper balance in contribution.

In [ ]:
# Create standard scaling for our X data
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

Xweight_train_scale = scaler.fit_transform(Xweight_train)
Xweight_test_scale = scaler.fit_transform(Xweight_test)

Xwaist_train_scale = scaler.fit_transform(Xwaist_train)
Xwaist_test_scale = scaler.fit_transform(Xwaist_test)

model_weight = Lasso()
model_weight.fit(Xweight_train_scale, yweight_train)
model_waist = Lasso()
model_waist.fit(Xwaist_train_scale, ywaist_train)

show_scores(model_weight, yweight_test, Xweight_test_scale, 'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test_scale, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(99.74639365713503),
   'Test MSE': 0.3663340273820337,
   'Test R^2': -4.741829151555521e-11}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.70550838500678),
   'Test MSE': 0.6185494940130513,
   'Test R^2': -5.9342357498382015e-05}))

Scaling results¶

Doesn't seem like there's much of an improvement with scaling the data. However the R^2 evaluation is 0, compared to our first model fit. Consulting with chatgpt, it states that the model is perfomring no better than baseline model, predicting the mean of our truth variables. Unable to pick up meaningful relationships between features and target variables.

Gridsearch CV¶

We'll use an adjustment tool, named Gridsearch CV. What it does is, I give it a list of different parameters with values, and the gridsearch does an exhaustive training list on all possible options, to find the best one.

In [ ]:
# Use gridsearch cv to find best parameters for Lasso()
from sklearn.model_selection import GridSearchCV

param_grid = {
    'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
    'max_iter':[1000,5000,10000,50000], # Iterations for convergence
    'tol':[1e-4,1e-3,1e-2,1e-1] # Tolerance for optimization
}

# weight grid search
grid_search_weight = GridSearchCV(
    estimator=Lasso(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

# waist grid search
grid_search_waist = GridSearchCV(
    estimator=Lasso(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)

Fitting 5 folds for each of 96 candidates, totalling 480 fits
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
Fitting 5 folds for each of 96 candidates, totalling 480 fits
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.930e+00, tolerance: 6.430e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.143e+00, tolerance: 8.334e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.930e+00, tolerance: 6.430e-01
  model = cd_fast.enet_coordinate_descent(

[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.124e-01, tolerance: 6.430e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.001e-01, tolerance: 6.430e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.2s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=Lasso(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'max_iter': [1000, 5000, 10000, 50000],
                          'tol': [0.0001, 0.001, 0.01, 0.1]},
              scoring='r2', verbose=2),
 GridSearchCV(cv=5, estimator=Lasso(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'max_iter': [1000, 5000, 10000, 50000],
                          'tol': [0.0001, 0.001, 0.01, 0.1]},
              scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(98.11454723696536),
   'Test MSE': 0.3556541293090464,
   'Test R^2': 0.029153442644805794}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(98.73011808148473),
   'Test MSE': 0.6193944786817607,
   'Test R^2': -0.0014254978879302271}))

Hmmm, it seems to provide little improvement in results...

I think out of curiousity sake, let's see what happens if we allow data leak, and shift the Weight and Waist difference back to it's original position, to see what kind of results come of it.

In [118]:
yweight_train = yweight_train.shift(1)
ywaist_train = ywaist_train.shift(1)

# fill nan at end of index with 0
ywaist_train = ywaist_train.fillna(0)
yweight_train = yweight_train.fillna(0)
ywaist_train, yweight_train
Out[118]:
(0       0.0
 1       0.2
 2       0.1
 3      -1.2
 4       0.3
        ... 
 1458    0.3
 1459    0.1
 1460    0.1
 1461   -1.0
 1462    0.5
 Name: Waist Difference, Length: 1463, dtype: float64,
 0       0.0
 1      -0.2
 2       0.6
 3      -0.6
 4      -0.3
        ... 
 1458    0.4
 1459   -0.4
 1460    0.1
 1461   -0.7
 1462    0.3
 Name: Weight Difference, Length: 1463, dtype: float64)
In [ ]:
# Retry GridSearchCV with leaked dataset
from sklearn.model_selection import GridSearchCV

param_grid = {
    'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
    'max_iter':[1000,5000,10000,50000], # Iterations for convergence
    'tol':[1e-4,1e-3,1e-2,1e-1] # Tolerance for optimization
}

# weight grid search
grid_search_weight = GridSearchCV(
    estimator=Lasso(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

# waist grid search
grid_search_waist = GridSearchCV(
    estimator=Lasso(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)

Fitting 5 folds for each of 96 candidates, totalling 480 fits
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
Fitting 5 folds for each of 96 candidates, totalling 480 fits
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.240e-01, tolerance: 8.334e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e-01
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e-01
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e+00
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e+00
  model = cd_fast.enet_coordinate_descent(

[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e-01
  model = cd_fast.enet_coordinate_descent(

[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e+00
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.751e+00, tolerance: 6.422e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.751e+00, tolerance: 6.422e-01
  model = cd_fast.enet_coordinate_descent(

[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.713e-01, tolerance: 6.422e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.4s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.3s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=1, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=10, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=10, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=1000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=1000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=5000, tol=0.001; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=5000, tol=0.01; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=10000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=10000, tol=0.1; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END ................alpha=100, max_iter=50000, tol=0.01; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
[CV] END .................alpha=100, max_iter=50000, tol=0.1; total time=   0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=Lasso(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'max_iter': [1000, 5000, 10000, 50000],
                          'tol': [0.0001, 0.001, 0.01, 0.1]},
              scoring='r2', verbose=2),
 GridSearchCV(cv=5, estimator=Lasso(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'max_iter': [1000, 5000, 10000, 50000],
                          'tol': [0.0001, 0.001, 0.01, 0.1]},
              scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(98.78643106399778),
   'Test MSE': 0.3448826776239478,
   'Test R^2': 0.058556803731916185}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(129.8633045684746),
   'Test MSE': 0.8208388707545833,
   'Test R^2': -0.32711705241647127}))

Well it seems to make the results worse as a matter of fact! I guess we'll shift it back to what it was before

In [119]:
yweight_train = yweight_train.shift(1)
ywaist_train = ywaist_train.shift(1)

# fill nan at end of index with 0
ywaist_train = ywaist_train.fillna(0)
yweight_train = yweight_train.fillna(0)

Model Trainig: ElasticNet¶

Time to try out ElasticNet, and see if it helps with model accuracy

In [ ]:
# replace lasso with elasticnet
from sklearn.linear_model import ElasticNet

model_weight = ElasticNet()
model_weight.fit(Xweight_train_scale, yweight_train)
model_waist = ElasticNet()
model_waist.fit(Xwaist_train_scale, ywaist_train)

show_scores(model_weight, yweight_test, Xweight_test_scale, 'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test_scale, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(99.7529524418643),
   'Test MSE': 0.36633419215872726,
   'Test R^2': -4.498464576396799e-07}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.69630552203824),
   'Test MSE': 0.6185537518838363,
   'Test R^2': -6.622640396258817e-05}))

Seems to be no different than Lasso when its fit on, right out of the bag. Let's try it with GridSearchCV, and see how well it compares to Lasso's GridSearchCV.

In [ ]:
# Try GridSearchCV with ElasticNet

param_grid = {
    'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
    'l1_ratio':[0.1,0.3,0.5,0.7,1] # Mix between Lasso and Ridge
}

# weight grid search
grid_search_weight = GridSearchCV(
    estimator=ElasticNet(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

# waist grid search
grid_search_waist = GridSearchCV(
    estimator=ElasticNet(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=2
)

grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)

Fitting 5 folds for each of 30 candidates, totalling 150 fits
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s

[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.623e+01, tolerance: 2.302e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.713e+01, tolerance: 2.572e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 5.096e+01, tolerance: 2.620e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.972e+01, tolerance: 2.192e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
Fitting 5 folds for each of 30 candidates, totalling 150 fits
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.3; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.5; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.730e+02, tolerance: 6.421e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.849e+00, tolerance: 7.045e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.347e+02, tolerance: 8.373e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.093e+02, tolerance: 8.786e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.978e+00, tolerance: 8.333e-02
  model = cd_fast.enet_coordinate_descent(
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.975e+01, tolerance: 6.421e-02
  model = cd_fast.enet_coordinate_descent(

[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.001, l1_ratio=1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.1; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.3; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.5; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END ...........................alpha=0.01, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=0.01, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=0.1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=0.1, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.1; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.3; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.5; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=1, l1_ratio=0.7; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END ................................alpha=1, l1_ratio=1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.1; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.3; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.5; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END .............................alpha=10, l1_ratio=0.7; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ...............................alpha=10, l1_ratio=1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.1; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.3; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.5; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ............................alpha=100, l1_ratio=0.7; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
[CV] END ..............................alpha=100, l1_ratio=1; total time=   0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=ElasticNet(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'l1_ratio': [0.1, 0.3, 0.5, 0.7, 1]},
              scoring='r2', verbose=2),
 GridSearchCV(cv=5, estimator=ElasticNet(),
              param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
                          'l1_ratio': [0.1, 0.3, 0.5, 0.7, 1]},
              scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(98.94022717664029),
   'Test MSE': 0.36639817179805406,
   'Test R^2': -0.00017509821255967672}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.69630552203824),
   'Test MSE': 0.6185537518838363,
   'Test R^2': -6.622640396258817e-05}))

Alright, something seems too fishy with the way things are playing out. Let's take a look out what values the model is predicting instead, and see if our hypothesis of the model predicting just 0s being correct.

In [ ]:
# have a peek at the prediction results
yweight_pred = grid_search_weight.best_estimator_.predict(Xweight_test)
ywaist_pred = grid_search_waist.best_estimator_.predict(Xwaist_test)

yweight_pred[:10], ywaist_pred[:10]
Out[ ]:
(array([0.03652816, 0.03547674, 0.03509221, 0.03596346, 0.03993248,
        0.04042726, 0.03794259, 0.03827334, 0.03912308, 0.03928173]),
 array([0.0112782, 0.0112782, 0.0112782, 0.0112782, 0.0112782, 0.0112782,
        0.0112782, 0.0112782, 0.0112782, 0.0112782]))

And why not plot it out for the heck of it as well with the true labels.

In [ ]:
fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
No description has been provided for this image

Oops, we broke the graph becuase the y test data still has the old index values. Let's remove it.

In [ ]:
# Reset index for y test data
yweight_test = yweight_test.reset_index(drop=True)
ywaist_test = ywaist_test.reset_index(drop=True)
In [ ]:
fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
No description has been provided for this image

Regardless, it's very clear that the ML model is not doing much in its way to try and predict the value, and it seems to be more so for the waist prediction, where the value just stays consistent all the way!

There must be an issue here with the way my data is set up. Maybe it's just because after all the values are averaged out, it hovers right at 0? Let's check out the mean.

In [120]:
# look at mean of weight and waist difference
print("Mean of Weight Difference:", yweight_train.mean())
print("Mean of Waist Difference:", ywaist_train.mean())

Mean of Weight Difference: 0.015447710184552287
Mean of Waist Difference: 0.011278195488721804

It is very close to 0. ChatGPT suggested, that maybe upscaling the difference can help the model learn, as too many values hovering right at 0, gives little to no opportunities for the model to learn.

So let's try multiplying our weight difference by 100

In [184]:
# multiply our weight and waist difference by 100
yweight_train = yweight_train*100
ywaist_train = ywaist_train*100
yweight_test = yweight_test*100
ywaist_test = ywaist_test*100

yweight_train, ywaist_test
Out[184]:
(0        0.0
 1        0.0
 2      -20.0
 3       60.0
 4      -60.0
         ... 
 1458    60.0
 1459    40.0
 1460   -40.0
 1461    10.0
 1462   -70.0
 Name: Weight Difference, Length: 1463, dtype: float64,
 1463     80.0
 1464     60.0
 1465    -20.0
 1466    -40.0
 1467     10.0
         ...  
 1622    100.0
 1623   -100.0
 1624      0.0
 1625      0.0
 1626      0.0
 Name: Waist Difference, Length: 164, dtype: float64)

Let's try it with the best estimators of ElasticNet

In [ ]:
# Try GridSearchCV with ElasticNet

param_grid = {
    'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
    'l1_ratio':[0.1,0.3,0.5,0.7,1] # Mix between Lasso and Ridge
}

# weight grid search
grid_search_weight = GridSearchCV(
    estimator=ElasticNet(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=1
)

# waist grid search
grid_search_waist = GridSearchCV(
    estimator=ElasticNet(),
    param_grid=param_grid,
    scoring='r2',
    cv=5,
    verbose=1
)

grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(98.99093421660329),
   'Test MSE': 3680.551799975127,
   'Test R^2': -0.004698314937412906}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(97.80167748816821),
   'Test MSE': 6221.224892910559,
   'Test R^2': -0.005836094812318837}))

Meh, didn't do any better. But let's still see it's prediction against the true value

In [ ]:
yweight_pred = grid_search_weight.best_estimator_.predict(Xweight_test)
ywaist_pred = grid_search_waist.best_estimator_.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
No description has been provided for this image

At least amplifying the difference by 100 seems to help just a little bit in having varied predictions. So it's time to try something else, and I figured that TensorFlow could be a good start!

Model Training: TensorFlow¶

What's really good with TensorFlow, is it can patterns in non-linear fashion, while ElasticNet learns its value linearly (which is not the case for our data).

Our problem is a regression problem, so ChatGPT suggests we have:

  • Have dense (fully connected) layers
  • Use an activation function that works well with regression
  • End with a single neuron output (for regression)

Activation functions:

  • Relu -> Best for hidden layers: helps with non-linearity
  • Linear -> Best for output layers: as there's no transformation needed
  • Adam Optimizer -> Good default choice for regression tasks: Adapts learning rates automatically, faster convergence, avoids getting stuck in plateaus, and works well with noisy data.

Let's create our first model, using Relu, Linear, and Adam Optimizer.

In [34]:
pip install tensorflow

Requirement already satisfied: tensorflow in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (2.18.0)
Requirement already satisfied: tensorflow-intel==2.18.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow) (2.18.0)
Requirement already satisfied: absl-py>=1.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.1.0)
Requirement already satisfied: astunparse>=1.6.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=24.3.25 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (25.2.10)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.6.0)
Requirement already satisfied: google-pasta>=0.1.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.2.0)
Requirement already satisfied: libclang>=13.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (18.1.1)
Requirement already satisfied: opt-einsum>=2.3.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.4.0)
Requirement already satisfied: packaging in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (24.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (5.29.3)
Requirement already satisfied: requests<3,>=2.21.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.32.3)
Requirement already satisfied: setuptools in c:\program files\windowsapps\pythonsoftwarefoundation.python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\lib\site-packages (from tensorflow-intel==2.18.0->tensorflow) (65.5.0)
Requirement already satisfied: six>=1.12.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.17.0)
Requirement already satisfied: termcolor>=1.1.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.5.0)
Requirement already satisfied: typing-extensions>=3.6.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (4.12.2)
Requirement already satisfied: wrapt>=1.11.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.17.2)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.70.0)
Requirement already satisfied: tensorboard<2.19,>=2.18 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.18.0)
Requirement already satisfied: keras>=3.5.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.8.0)
Requirement already satisfied: numpy<2.1.0,>=1.26.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.0.2)
Requirement already satisfied: h5py>=3.11.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.12.1)
Requirement already satisfied: ml-dtypes<0.5.0,>=0.4.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.4.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.31.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from astunparse>=1.6.0->tensorflow-intel==2.18.0->tensorflow) (0.45.1)
Requirement already satisfied: rich in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (13.9.4)
Requirement already satisfied: namex in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.0.8)
Requirement already satisfied: optree in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.14.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (2025.1.31)
Requirement already satisfied: markdown>=2.6.8 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.7)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.1.3)
Requirement already satisfied: MarkupSafe>=2.1.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from werkzeug>=1.0.1->tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.0.2)
Requirement already satisfied: markdown-it-py>=2.2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (2.19.1)
Requirement already satisfied: mdurl~=0.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.1.2)
Note: you may need to restart the kernel to use updated packages.

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip
In [35]:
import tensorflow as tf

MODEL 1¶

In [37]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_1 = keras.Sequential([
    layers.Dense(1, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(1, activation='linear')
])

model_1.compile(optimizer='adam',
                loss='mape',
                metrics=['mae'])

modelweight_1 = model_1
modelwaist_1 = model_1
modelweight_1.fit(Xweight_train, yweight_train, epochs=100), modelwaist_1.fit(Xwaist_train, ywaist_train, epochs=100)

Epoch 1/100

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

46/46 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 814765760.0000 - mae: 4.6365
Epoch 2/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50534888.0000 - mae: 0.5537
Epoch 3/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28712.1465 - mae: 0.4294
Epoch 4/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 34377.3086 - mae: 0.4289
Epoch 5/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18265.8555 - mae: 0.4289 
Epoch 6/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 47448.5586 - mae: 0.4289
Epoch 7/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69117.7266 - mae: 0.4290
Epoch 8/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 21261.1113 - mae: 0.4289 
Epoch 9/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 30248.6875 - mae: 0.4289
Epoch 10/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47582.1367 - mae: 0.4289 
Epoch 11/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69111.7656 - mae: 0.4290
Epoch 12/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 21224.6484 - mae: 0.4289 
Epoch 13/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26699.9727 - mae: 0.4289
Epoch 14/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27917.8438 - mae: 0.4289 
Epoch 15/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31166.2012 - mae: 0.4289 
Epoch 16/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14923.9990 - mae: 0.4289 
Epoch 17/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18820.5586 - mae: 0.4289 
Epoch 18/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27072.0430 - mae: 0.4289 
Epoch 19/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26369.6387 - mae: 0.4289
Epoch 20/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50906.2383 - mae: 0.4290 
Epoch 21/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47244.7109 - mae: 0.4289 
Epoch 22/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26295.0137 - mae: 0.4289
Epoch 23/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 34885.7734 - mae: 0.4289 
Epoch 24/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 36128.2148 - mae: 0.4289 
Epoch 25/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50980.9727 - mae: 0.4290 
Epoch 26/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14363.5986 - mae: 0.4289 
Epoch 27/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 35372.1680 - mae: 0.4289
Epoch 28/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29165.3652 - mae: 0.4289 
Epoch 29/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26405.4590 - mae: 0.4289
Epoch 30/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50964.0859 - mae: 0.4290 
Epoch 31/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47349.0234 - mae: 0.4289 
Epoch 32/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26348.5039 - mae: 0.4289
Epoch 33/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 34790.0625 - mae: 0.4289 
Epoch 34/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 36128.8906 - mae: 0.4289 
Epoch 35/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50901.3320 - mae: 0.4290 
Epoch 36/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14401.1436 - mae: 0.4289 
Epoch 37/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 19837.3184 - mae: 0.4289
Epoch 38/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47385.5586 - mae: 0.4290 
Epoch 39/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69208.2109 - mae: 0.4290
Epoch 40/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 22232.1621 - mae: 0.4289 
Epoch 41/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32560.6211 - mae: 0.4289 
Epoch 42/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 9653.5391 - mae: 0.4289  
Epoch 43/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28403.5703 - mae: 0.4289 
Epoch 44/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 33961.5703 - mae: 0.4289 
Epoch 45/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29111.8223 - mae: 0.4289
Epoch 46/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24103.3867 - mae: 0.4289
Epoch 47/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 12425.9824 - mae: 0.4289 
Epoch 48/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 33108.3711 - mae: 0.4289 
Epoch 49/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92331.6953 - mae: 0.4290 
Epoch 50/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24899.2012 - mae: 0.4289
Epoch 51/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29133.5195 - mae: 0.4289
Epoch 52/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94018.1641 - mae: 0.4290 
Epoch 53/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31684.0586 - mae: 0.4289 
Epoch 54/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.7920 - mae: 0.4289 
Epoch 55/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32233.0938 - mae: 0.4289 
Epoch 56/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29290.6699 - mae: 0.4289
Epoch 57/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92252.6641 - mae: 0.4290 
Epoch 58/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.9785 - mae: 0.4289
Epoch 59/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29132.3145 - mae: 0.4289
Epoch 60/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94016.2969 - mae: 0.4290 
Epoch 61/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.7598 - mae: 0.4289 
Epoch 62/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.8213 - mae: 0.4289 
Epoch 63/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32231.5957 - mae: 0.4289 
Epoch 64/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29289.7793 - mae: 0.4289
Epoch 65/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92250.4297 - mae: 0.4290 
Epoch 66/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7910 - mae: 0.4289
Epoch 67/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29131.5098 - mae: 0.4289
Epoch 68/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.3047 - mae: 0.4290 
Epoch 69/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.0977 - mae: 0.4289 
Epoch 70/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.9258 - mae: 0.4289 
Epoch 71/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32230.4707 - mae: 0.4289 
Epoch 72/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29289.2168 - mae: 0.4289
Epoch 73/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 92248.6719 - mae: 0.4290
Epoch 74/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7344 - mae: 0.4289
Epoch 75/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.9590 - mae: 0.4289
Epoch 76/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94014.9609 - mae: 0.4290 
Epoch 77/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31681.8516 - mae: 0.4289 
Epoch 78/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.0859 - mae: 0.4289 
Epoch 79/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32229.6016 - mae: 0.4289 
Epoch 80/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.8418 - mae: 0.4289
Epoch 81/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92247.2109 - mae: 0.4290 
Epoch 82/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7559 - mae: 0.4289
Epoch 83/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.5742 - mae: 0.4289
Epoch 84/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94014.9766 - mae: 0.4290 
Epoch 85/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31681.8750 - mae: 0.4289 
Epoch 86/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.2803 - mae: 0.4289 
Epoch 87/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32228.8945 - mae: 0.4289 
Epoch 88/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.6172 - mae: 0.4289
Epoch 89/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92245.9766 - mae: 0.4290 
Epoch 90/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.8242 - mae: 0.4289
Epoch 91/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.3047 - mae: 0.4289
Epoch 92/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.2344 - mae: 0.4290 
Epoch 93/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.0371 - mae: 0.4289 
Epoch 94/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.4717 - mae: 0.4289 
Epoch 95/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32228.3145 - mae: 0.4289 
Epoch 96/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.4707 - mae: 0.4289
Epoch 97/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92244.8906 - mae: 0.4290 
Epoch 98/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.9199 - mae: 0.4289
Epoch 99/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.0918 - mae: 0.4289
Epoch 100/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.6250 - mae: 0.4290 
Epoch 1/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 51684.5547 - mae: 0.6845 
Epoch 2/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 37540.6523 - mae: 0.6845 
Epoch 3/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 10654.8145 - mae: 0.6845
Epoch 4/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29097.0879 - mae: 0.6845 
Epoch 5/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18858.0938 - mae: 0.6845 
Epoch 6/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18007.2969 - mae: 0.6845 
Epoch 7/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 19149.3672 - mae: 0.6845 
Epoch 8/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 35210.2695 - mae: 0.6845 
Epoch 9/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 38894.6602 - mae: 0.6845
Epoch 10/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28572.7109 - mae: 0.6845 
Epoch 11/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18018.6562 - mae: 0.6845 
Epoch 12/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18118.5293 - mae: 0.6845 
Epoch 13/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27053.2246 - mae: 0.6845
Epoch 14/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47434.3789 - mae: 0.6845 
Epoch 15/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 22458.3887 - mae: 0.6845 
Epoch 16/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 17581.8223 - mae: 0.6845 
Epoch 17/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47249.5391 - mae: 0.6845 
Epoch 18/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60501.5117 - mae: 0.6845 
Epoch 19/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60025.2148 - mae: 0.6845 
Epoch 20/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61765.7344 - mae: 0.6845
Epoch 21/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60195.4336 - mae: 0.6845 
Epoch 22/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61936.9453 - mae: 0.6845
Epoch 23/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 60359.7031 - mae: 0.6845
Epoch 24/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62094.1875 - mae: 0.6845
Epoch 25/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60510.5898 - mae: 0.6845 
Epoch 26/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62238.5469 - mae: 0.6845
Epoch 27/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60649.1172 - mae: 0.6845 
Epoch 28/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62371.0273 - mae: 0.6845
Epoch 29/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60776.2383 - mae: 0.6845 
Epoch 30/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62492.5547 - mae: 0.6845
Epoch 31/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60892.8125 - mae: 0.6845 
Epoch 32/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62603.9414 - mae: 0.6845
Epoch 33/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60999.6836 - mae: 0.6845 
Epoch 34/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62706.0312 - mae: 0.6845
Epoch 35/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61097.5977 - mae: 0.6845 
Epoch 36/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62799.5312 - mae: 0.6845
Epoch 37/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61187.2891 - mae: 0.6845 
Epoch 38/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62885.1328 - mae: 0.6845
Epoch 39/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61269.4141 - mae: 0.6844 
Epoch 40/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62963.4805 - mae: 0.6845
Epoch 41/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61344.5898 - mae: 0.6844 
Epoch 42/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63035.1680 - mae: 0.6845
Epoch 43/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61413.3750 - mae: 0.6844 
Epoch 44/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63100.7383 - mae: 0.6845
Epoch 45/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61476.3242 - mae: 0.6844 
Epoch 46/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63160.6953 - mae: 0.6844
Epoch 47/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61533.8945 - mae: 0.6844 
Epoch 48/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63215.5195 - mae: 0.6844
Epoch 49/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61586.5273 - mae: 0.6844 
Epoch 50/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63265.6172 - mae: 0.6844
Epoch 51/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61634.6523 - mae: 0.6844 
Epoch 52/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63311.4102 - mae: 0.6844
Epoch 53/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61678.6328 - mae: 0.6844 
Epoch 54/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63353.2422 - mae: 0.6844
Epoch 55/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61718.8359 - mae: 0.6844 
Epoch 56/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63391.4492 - mae: 0.6844
Epoch 57/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61755.5664 - mae: 0.6844 
Epoch 58/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63426.3398 - mae: 0.6844
Epoch 59/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 61789.1445 - mae: 0.6843 
Epoch 60/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63458.2070 - mae: 0.6844
Epoch 61/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61819.8086 - mae: 0.6843 
Epoch 62/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63487.2969 - mae: 0.6843
Epoch 63/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61847.8516 - mae: 0.6843 
Epoch 64/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63513.8555 - mae: 0.6843
Epoch 65/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61873.4414 - mae: 0.6843 
Epoch 66/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63538.0781 - mae: 0.6843
Epoch 67/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61896.8164 - mae: 0.6843 
Epoch 68/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63560.2070 - mae: 0.6843
Epoch 69/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61918.1719 - mae: 0.6843 
Epoch 70/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63580.3945 - mae: 0.6843
Epoch 71/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 61937.6836 - mae: 0.6842
Epoch 72/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63598.7969 - mae: 0.6842
Epoch 73/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61955.5039 - mae: 0.6842 
Epoch 74/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63615.5977 - mae: 0.6842
Epoch 75/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61971.7734 - mae: 0.6842 
Epoch 76/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63630.9180 - mae: 0.6842
Epoch 77/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61986.6133 - mae: 0.6842 
Epoch 78/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63644.8828 - mae: 0.6842
Epoch 79/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62000.1719 - mae: 0.6841 
Epoch 80/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63657.6211 - mae: 0.6841
Epoch 81/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62012.5703 - mae: 0.6841 
Epoch 82/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63669.2305 - mae: 0.6841
Epoch 83/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62023.8750 - mae: 0.6841 
Epoch 84/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63679.8203 - mae: 0.6841
Epoch 85/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62034.2148 - mae: 0.6840 
Epoch 86/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63689.4570 - mae: 0.6840
Epoch 87/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62043.6484 - mae: 0.6840 
Epoch 88/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63698.2656 - mae: 0.6840
Epoch 89/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62052.2656 - mae: 0.6840 
Epoch 90/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63706.2891 - mae: 0.6840
Epoch 91/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62060.1328 - mae: 0.6839 
Epoch 92/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63713.5898 - mae: 0.6839
Epoch 93/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62067.3242 - mae: 0.6839 
Epoch 94/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63720.2305 - mae: 0.6839
Epoch 95/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62073.8789 - mae: 0.6838 
Epoch 96/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63726.2891 - mae: 0.6838
Epoch 97/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62079.8828 - mae: 0.6838 
Epoch 98/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63731.7891 - mae: 0.6838
Epoch 99/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62085.3672 - mae: 0.6837
Epoch 100/100
46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63736.8125 - mae: 0.6837
Out[37]:
(<keras.src.callbacks.history.History at 0x1aecf17a710>,
 <keras.src.callbacks.history.History at 0x1aecf172f90>)
In [41]:
show_scores(modelweight_1, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_1, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[41]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(475.9956588090121),
   'Test MSE': 168.39310068358228,
   'Test R^2': -458.6709235420203}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(195.89982110985835),
   'Test MSE': 168.02178535139865,
   'Test R^2': -270.6545042662456}))

Tensorflow 2D array¶

One thing different with Lasso and ElasticNet, is it returns predictions in a 1D array such as ((n,)) compared to TensorFlow that make predictions in 2D array such as (n,1).

To fix this, we will need to revise the show_scores function, and use the np.squeeze() function inside it, to set the array as a 1D.

In [39]:
# Create MAPE evaluation metric from numpy
def mean_absolute_percent_error(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    dif = np.abs(y_true - y_pred)

    percent_error = np.zeros_like(y_true, dtype=float)

    # flatten tensor array
    y_pred = np.squeeze(y_pred)

    # Task1 > If true is 0, and pred is not 0, set to 100%
    percent_error[(y_true == 0) & (y_pred != 0)] = 100

    # Task2 > If true and pred is 0, set to 0%
    percent_error[(y_true == 0) & (y_pred == 0)] = 0

    # Task3 > All other cases are then calculated
    mask = (y_true != 0) # boolean array of where y_true is 0 or not
    percent_error[mask] = (np.abs(y_true[mask] - y_pred[mask]) / np.abs(y_true[mask])) * 100 # absolute dif of true and pred, divided by true. Multiply bvy 100 to get percentage

    return np.mean(percent_error)


from sklearn.metrics import r2_score
# also add in mse. r^2 is built in with model scoring
from sklearn.metrics import mean_squared_error

# create function to include all evaluation metric in one space when testing models
def show_scores(model, truth, test, title):
    
    # model predicting values from training
    preds = model.predict(test)

    # calculating scores
    scores = {
        "Test MAPE (%)":mean_absolute_percent_error(truth, preds),
        "Test MSE":mean_squared_error(truth, preds),
        "Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y
    }

    return title, scores
In [42]:
show_scores(modelweight_1, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_1, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[42]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(475.9956588090121),
   'Test MSE': 168.39310068358228,
   'Test R^2': -458.6709235420203}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(195.89982110985835),
   'Test MSE': 168.02178535139865,
   'Test R^2': -270.6545042662456}))

MODEL 2¶

Not good. Let's up our mode layers with more neurons. Increasing by the power of 2 (1, 2, 4, 8, 16, 32)

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_2 = keras.Sequential([
    layers.Dense(1, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(2, activation='relu'),
    layers.Dense(4, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

model_2.compile(optimizer='adam',
                loss='mape',
                metrics=['mae'])

modelweight_2 = model_2
modelwaist_2 = model_2
modelweight_2.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_2.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd3590b6d0>,
 <keras.src.callbacks.history.History at 0x2bd35879090>)
In [ ]:
show_scores(modelweight_2, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_2, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(99.99998191581312),
   'Test MSE': 3665.918995282517,
   'Test R^2': -0.0007039263194963397}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.99996955052484),
   'Test MSE': 6185.43143862701,
   'Test R^2': -4.9075552263522226e-05}))

MODEL 3¶

Again there's no improvement. How about we reverse the exponent on relu? going 8,4,2,1 instead. We'll also remove the 1 for the relu layer, as it's not helpful to have one singular neuron layer, when there is already one existing specifically for the last layer linear.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_3 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(8, activation='relu'),
    layers.Dense(4, activation='relu'),
    layers.Dense(2, activation='relu'),
    layers.Dense(1, activation='linear')
])

model_3.compile(optimizer='adam',
                loss='mape',
                metrics=['mae'])

modelweight_3 = model_3
modelwaist_3 = model_3
modelweight_3.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_3.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37836190>,
 <keras.src.callbacks.history.History at 0x2bd37c25050>)
In [ ]:
show_scores(modelweight_3, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_3, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(99.99996217106948),
   'Test MSE': 3665.8529085824352,
   'Test R^2': -0.000685886308155359}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.99993630506658),
   'Test MSE': 6185.365622942159,
   'Test R^2': -3.843459448304998e-05}))

MODEL 4¶

Let's see if changing the loss helps. Maybe I'm too over-reliant on my fabricated function, that it's not working for the data specifically. Let's switch it to MSE

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_4 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(8, activation='relu'),
    layers.Dense(4, activation='relu'),
    layers.Dense(2, activation='relu'),
    layers.Dense(1, activation='linear')
])

model_4.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

modelweight_4 = model_4
modelwaist_4 = model_4
modelweight_4.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_4.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37cb7f50>,
 <keras.src.callbacks.history.History at 0x2bd379b1f50>)
In [ ]:
show_scores(modelweight_4, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_4, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(99.76027602236368),
   'Test MSE': 3672.847703900664,
   'Test R^2': -0.0025952899659449358}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.59636176379766),
   'Test MSE': 6195.650336744256,
   'Test R^2': -0.0017012480347673442}))

MODEL 5¶

Now it looks like it's slightly helpful, but still barely any improvement.

Let's increase neurons in relu, starting from 64

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_5 = keras.Sequential([
    layers.Dense(64, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

model_5.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

modelweight_5 = model_5
modelwaist_5 = model_5
modelweight_5.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_5.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37d314d0>,
 <keras.src.callbacks.history.History at 0x2bd395a0a50>)
In [ ]:
show_scores(modelweight_5, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_5, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(115.64029176907775),
   'Test MSE': 4204.008713731997,
   'Test R^2': -0.1475889215028252}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(105.77201532508138),
   'Test MSE': 6786.071699910744,
   'Test R^2': -0.09715947827780091}))

Nevermind, this code performed worse for some reason.

However, I figure it'd be a good time to compare the graph and see what is going on

In [ ]:
yweight_pred = modelweight_5.predict(Xweight_test)
ywaist_pred = modelwaist_5.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
No description has been provided for this image
In [ ]:
yweight_pred = modelweight_4.predict(Xweight_test)
ywaist_pred = modelwaist_4.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
No description has been provided for this image

Okay, interesting results! Despite seeing poorer results for increasing the number of neurons in the model, it has the effect of the ML model trying to effectively learn the pattern of the data, over a simple straight line over 0. Sure we can more easily achieve a low score through getting the average which is close to 0, but we need that risk taking to catch the full dataset!

MODEL 6¶

I remember from TensorFlow 01 extra cirriculars, I did something where I increased exponentially in neurons, then decreased exponentially. Let's give it a go, and increase the top to 256 neurons. (16, 32, 64, 128, 256, 128, 64, 32, 16)

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
model_6 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

model_6.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

modelweight_6 = model_6
modelwaist_6 = model_6
modelweight_6.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_6.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd3ad5ccd0>,
 <keras.src.callbacks.history.History at 0x2bd3c075050>)
In [ ]:
show_scores(modelweight_6, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_6, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
  {'Test MAPE (%)': np.float64(111.76415878255344),
   'Test MSE': 4159.561606060438,
   'Test R^2': -0.13545597606193827}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(93.5839074131618),
   'Test MSE': 6428.471201429294,
   'Test R^2': -0.03934329334846254}))
In [ ]:
yweight_pred = modelweight_6.predict(Xweight_test)
ywaist_pred = modelwaist_6.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
No description has been provided for this image

Increasing the number of neurons for ReLU, does seem to improve the variability in the prediction, which is our goal for now. We need the model to take risks in making some extreme predictions.

Weight and Waist same prediciton¶

One other thing, is I tried to train the model with the same layers, and figured that naming them differently will get them to train differently between the two. This doesn't seem to happen in our predictions, as they're both the same. Let's see if we can fix this by separating them into different codes.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelweight_6 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelweight_6.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

modelweight_6.fit(Xweight_train, yweight_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd3d511610>
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_6 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelwaist_6.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

modelwaist_6.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd526a1a50>
In [ ]:
yweight_pred = modelweight_6.predict(Xweight_test)
ywaist_pred = modelwaist_6.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step
No description has been provided for this image

Sweet, we got them to both to separate, by training them on separate blocks of code. For now, we'll train on the waist difference data, as both weight and waist have very similar variables and patterns. So theoretically, the model that works well in predicting my waist difference, should work on weight difference as well if I retrain the model on the weight data.

MODEL 7¶

Next step, let's try find the optimal Adam optimizer value. To do so, we'll make a Learning Rate Scheduler, where we exponentially increase the learning rate of Adam. We do this, as learning rate doesn't increase linearly like 0.01, 0.02, 0.03, but exponentially as 0.001, 0.01, 0.1.

In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)

adam_model = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

adam_model.compile(optimizer='adam',
                loss='mse',
                metrics=['mae'])

# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase

find_lr_history = adam_model.fit(Xwaist_train, 
                                   ywaist_train, 
                                   epochs=100, 
                                   verbose=2,
                                   validation_data=(Xwaist_test,ywaist_test),
                                   callbacks=[lr_scheduler])

Epoch 1/100

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

46/46 - 3s - 58ms/step - loss: 6685.2480 - mae: 58.8719 - val_loss: 6198.9521 - val_mae: 56.1354 - learning_rate: 1.0000e-05
Epoch 2/100
46/46 - 0s - 4ms/step - loss: 6672.3242 - mae: 58.8637 - val_loss: 6194.2646 - val_mae: 56.0520 - learning_rate: 1.1220e-05
Epoch 3/100
46/46 - 0s - 4ms/step - loss: 6671.0337 - mae: 58.8408 - val_loss: 6198.7007 - val_mae: 56.1022 - learning_rate: 1.2589e-05
Epoch 4/100
46/46 - 0s - 4ms/step - loss: 6671.8237 - mae: 58.8759 - val_loss: 6206.0981 - val_mae: 56.1709 - learning_rate: 1.4125e-05
Epoch 5/100
46/46 - 0s - 4ms/step - loss: 6672.6294 - mae: 58.9178 - val_loss: 6212.1694 - val_mae: 56.2194 - learning_rate: 1.5849e-05
Epoch 6/100
46/46 - 0s - 4ms/step - loss: 6673.7681 - mae: 58.9777 - val_loss: 6232.3481 - val_mae: 56.3535 - learning_rate: 1.7783e-05
Epoch 7/100
46/46 - 0s - 4ms/step - loss: 6676.5757 - mae: 59.0470 - val_loss: 6236.3271 - val_mae: 56.4014 - learning_rate: 1.9953e-05
Epoch 8/100
46/46 - 0s - 4ms/step - loss: 6673.7373 - mae: 59.0563 - val_loss: 6255.1958 - val_mae: 56.5190 - learning_rate: 2.2387e-05
Epoch 9/100
46/46 - 0s - 4ms/step - loss: 6675.2412 - mae: 59.1204 - val_loss: 6283.3394 - val_mae: 56.7068 - learning_rate: 2.5119e-05
Epoch 10/100
46/46 - 0s - 4ms/step - loss: 6679.2373 - mae: 59.2138 - val_loss: 6295.4517 - val_mae: 56.8270 - learning_rate: 2.8184e-05
Epoch 11/100
46/46 - 0s - 4ms/step - loss: 6679.3535 - mae: 59.2373 - val_loss: 6314.8853 - val_mae: 57.0322 - learning_rate: 3.1623e-05
Epoch 12/100
46/46 - 0s - 4ms/step - loss: 6680.6235 - mae: 59.2825 - val_loss: 6329.9893 - val_mae: 57.2232 - learning_rate: 3.5481e-05
Epoch 13/100
46/46 - 0s - 4ms/step - loss: 6677.6577 - mae: 59.2895 - val_loss: 6348.1636 - val_mae: 57.4229 - learning_rate: 3.9811e-05
Epoch 14/100
46/46 - 0s - 4ms/step - loss: 6674.8618 - mae: 59.3052 - val_loss: 6360.3604 - val_mae: 57.5080 - learning_rate: 4.4668e-05
Epoch 15/100
46/46 - 0s - 3ms/step - loss: 6669.9160 - mae: 59.2418 - val_loss: 6337.0190 - val_mae: 57.3272 - learning_rate: 5.0119e-05
Epoch 16/100
46/46 - 0s - 7ms/step - loss: 6677.8647 - mae: 59.3522 - val_loss: 6286.3984 - val_mae: 56.8164 - learning_rate: 5.6234e-05
Epoch 17/100
46/46 - 0s - 4ms/step - loss: 6688.3652 - mae: 59.3248 - val_loss: 6215.1484 - val_mae: 56.2655 - learning_rate: 6.3096e-05
Epoch 18/100
46/46 - 0s - 4ms/step - loss: 6688.8496 - mae: 59.1789 - val_loss: 6219.3101 - val_mae: 56.2795 - learning_rate: 7.0795e-05
Epoch 19/100
46/46 - 0s - 4ms/step - loss: 6683.4482 - mae: 59.1189 - val_loss: 6200.3442 - val_mae: 56.0701 - learning_rate: 7.9433e-05
Epoch 20/100
46/46 - 0s - 4ms/step - loss: 6684.1421 - mae: 59.2445 - val_loss: 6221.9268 - val_mae: 56.3175 - learning_rate: 8.9125e-05
Epoch 21/100
46/46 - 0s - 4ms/step - loss: 6672.0439 - mae: 58.9549 - val_loss: 6194.8135 - val_mae: 55.9934 - learning_rate: 1.0000e-04
Epoch 22/100
46/46 - 0s - 4ms/step - loss: 6662.1108 - mae: 58.8144 - val_loss: 6192.9604 - val_mae: 55.9525 - learning_rate: 1.1220e-04
Epoch 23/100
46/46 - 0s - 4ms/step - loss: 6660.2031 - mae: 58.7288 - val_loss: 6193.5547 - val_mae: 55.9644 - learning_rate: 1.2589e-04
Epoch 24/100
46/46 - 0s - 4ms/step - loss: 6660.9663 - mae: 58.7289 - val_loss: 6192.8857 - val_mae: 55.9468 - learning_rate: 1.4125e-04
Epoch 25/100
46/46 - 0s - 4ms/step - loss: 6661.2769 - mae: 58.7287 - val_loss: 6196.5107 - val_mae: 56.0061 - learning_rate: 1.5849e-04
Epoch 26/100
46/46 - 0s - 3ms/step - loss: 6663.8188 - mae: 58.7561 - val_loss: 6195.3076 - val_mae: 55.9926 - learning_rate: 1.7783e-04
Epoch 27/100
46/46 - 0s - 3ms/step - loss: 6664.7349 - mae: 58.7807 - val_loss: 6198.1777 - val_mae: 56.0418 - learning_rate: 1.9953e-04
Epoch 28/100
46/46 - 0s - 4ms/step - loss: 6665.1123 - mae: 58.8003 - val_loss: 6199.1777 - val_mae: 56.0512 - learning_rate: 2.2387e-04
Epoch 29/100
46/46 - 0s - 4ms/step - loss: 6664.9165 - mae: 58.8081 - val_loss: 6203.6729 - val_mae: 56.1273 - learning_rate: 2.5119e-04
Epoch 30/100
46/46 - 0s - 4ms/step - loss: 6666.5435 - mae: 58.8500 - val_loss: 6189.1016 - val_mae: 55.8678 - learning_rate: 2.8184e-04
Epoch 31/100
46/46 - 0s - 4ms/step - loss: 6675.3506 - mae: 58.9821 - val_loss: 6185.7739 - val_mae: 55.6677 - learning_rate: 3.1623e-04
Epoch 32/100
46/46 - 0s - 4ms/step - loss: 6665.8774 - mae: 58.6488 - val_loss: 6200.5791 - val_mae: 56.2484 - learning_rate: 3.5481e-04
Epoch 33/100
46/46 - 0s - 3ms/step - loss: 6671.8237 - mae: 58.8886 - val_loss: 6224.8320 - val_mae: 56.3359 - learning_rate: 3.9811e-04
Epoch 34/100
46/46 - 0s - 4ms/step - loss: 6669.1304 - mae: 58.8822 - val_loss: 6185.3228 - val_mae: 55.7860 - learning_rate: 4.4668e-04
Epoch 35/100
46/46 - 0s - 4ms/step - loss: 6663.7876 - mae: 58.7177 - val_loss: 6186.2969 - val_mae: 55.8232 - learning_rate: 5.0119e-04
Epoch 36/100
46/46 - 0s - 4ms/step - loss: 6659.3828 - mae: 58.6457 - val_loss: 6183.6606 - val_mae: 55.7271 - learning_rate: 5.6234e-04
Epoch 37/100
46/46 - 0s - 4ms/step - loss: 6667.7988 - mae: 58.6138 - val_loss: 6185.0737 - val_mae: 55.7521 - learning_rate: 6.3096e-04
Epoch 38/100
46/46 - 0s - 4ms/step - loss: 6663.3281 - mae: 58.6729 - val_loss: 6186.1538 - val_mae: 55.8061 - learning_rate: 7.0795e-04
Epoch 39/100
46/46 - 0s - 4ms/step - loss: 6661.1553 - mae: 58.7320 - val_loss: 6186.3018 - val_mae: 55.8084 - learning_rate: 7.9433e-04
Epoch 40/100
46/46 - 0s - 4ms/step - loss: 6660.1353 - mae: 58.7124 - val_loss: 6186.2378 - val_mae: 55.8017 - learning_rate: 8.9125e-04
Epoch 41/100
46/46 - 0s - 4ms/step - loss: 6660.4951 - mae: 58.6951 - val_loss: 6185.3828 - val_mae: 55.7754 - learning_rate: 1.0000e-03
Epoch 42/100
46/46 - 0s - 4ms/step - loss: 6660.5625 - mae: 58.6947 - val_loss: 6185.5791 - val_mae: 55.7824 - learning_rate: 0.0011
Epoch 43/100
46/46 - 0s - 3ms/step - loss: 6659.8203 - mae: 58.6846 - val_loss: 6185.5142 - val_mae: 55.7753 - learning_rate: 0.0013
Epoch 44/100
46/46 - 0s - 4ms/step - loss: 6659.0610 - mae: 58.6676 - val_loss: 6185.7871 - val_mae: 55.7912 - learning_rate: 0.0014
Epoch 45/100
46/46 - 0s - 3ms/step - loss: 6659.4551 - mae: 58.6649 - val_loss: 6185.5303 - val_mae: 55.7794 - learning_rate: 0.0016
Epoch 46/100
46/46 - 0s - 4ms/step - loss: 6659.5181 - mae: 58.6710 - val_loss: 6185.6777 - val_mae: 55.7965 - learning_rate: 0.0018
Epoch 47/100
46/46 - 0s - 4ms/step - loss: 6659.9595 - mae: 58.6583 - val_loss: 6185.2598 - val_mae: 55.7675 - learning_rate: 0.0020
Epoch 48/100
46/46 - 0s - 4ms/step - loss: 6659.7246 - mae: 58.6322 - val_loss: 6184.7739 - val_mae: 55.7363 - learning_rate: 0.0022
Epoch 49/100
46/46 - 0s - 3ms/step - loss: 6657.2500 - mae: 58.6726 - val_loss: 6521.6450 - val_mae: 59.5835 - learning_rate: 0.0025
Epoch 50/100
46/46 - 0s - 4ms/step - loss: 6925.8135 - mae: 60.5797 - val_loss: 6185.8862 - val_mae: 55.9682 - learning_rate: 0.0028
Epoch 51/100
46/46 - 0s - 4ms/step - loss: 6668.2227 - mae: 58.8161 - val_loss: 6182.8013 - val_mae: 55.9339 - learning_rate: 0.0032
Epoch 52/100
46/46 - 0s - 4ms/step - loss: 6694.4438 - mae: 58.9545 - val_loss: 6186.1021 - val_mae: 55.7800 - learning_rate: 0.0035
Epoch 53/100
46/46 - 0s - 4ms/step - loss: 6658.6919 - mae: 58.4989 - val_loss: 6185.0527 - val_mae: 55.6605 - learning_rate: 0.0040
Epoch 54/100
46/46 - 0s - 4ms/step - loss: 6663.8101 - mae: 58.6106 - val_loss: 6185.4736 - val_mae: 55.6143 - learning_rate: 0.0045
Epoch 55/100
46/46 - 0s - 3ms/step - loss: 6673.8262 - mae: 58.7408 - val_loss: 6186.1309 - val_mae: 55.7030 - learning_rate: 0.0050
Epoch 56/100
46/46 - 0s - 4ms/step - loss: 6659.2207 - mae: 58.5080 - val_loss: 6184.7388 - val_mae: 55.6649 - learning_rate: 0.0056
Epoch 57/100
46/46 - 0s - 4ms/step - loss: 6659.6392 - mae: 58.5433 - val_loss: 6185.1011 - val_mae: 55.6567 - learning_rate: 0.0063
Epoch 58/100
46/46 - 0s - 4ms/step - loss: 6667.4302 - mae: 58.5852 - val_loss: 6181.9668 - val_mae: 55.7025 - learning_rate: 0.0071
Epoch 59/100
46/46 - 0s - 4ms/step - loss: 6660.6426 - mae: 58.5344 - val_loss: 6185.2515 - val_mae: 55.6256 - learning_rate: 0.0079
Epoch 60/100
46/46 - 0s - 4ms/step - loss: 6659.3555 - mae: 58.5379 - val_loss: 6184.2944 - val_mae: 55.7058 - learning_rate: 0.0089
Epoch 61/100
46/46 - 0s - 4ms/step - loss: 6660.7383 - mae: 58.5535 - val_loss: 6185.3726 - val_mae: 55.6192 - learning_rate: 0.0100
Epoch 62/100
46/46 - 0s - 4ms/step - loss: 6669.2344 - mae: 58.6457 - val_loss: 6185.4141 - val_mae: 55.6365 - learning_rate: 0.0112
Epoch 63/100
46/46 - 0s - 4ms/step - loss: 6745.3027 - mae: 59.4220 - val_loss: 6185.3462 - val_mae: 55.6509 - learning_rate: 0.0126
Epoch 64/100
46/46 - 0s - 4ms/step - loss: 7612.5645 - mae: 61.5971 - val_loss: 6185.1729 - val_mae: 55.6866 - learning_rate: 0.0141
Epoch 65/100
46/46 - 0s - 4ms/step - loss: 6658.2686 - mae: 58.5969 - val_loss: 6185.2148 - val_mae: 55.6957 - learning_rate: 0.0158
Epoch 66/100
46/46 - 0s - 4ms/step - loss: 6658.2686 - mae: 58.6081 - val_loss: 6185.2568 - val_mae: 55.7027 - learning_rate: 0.0178
Epoch 67/100
46/46 - 0s - 4ms/step - loss: 6658.2959 - mae: 58.6179 - val_loss: 6185.3013 - val_mae: 55.7090 - learning_rate: 0.0200
Epoch 68/100
46/46 - 0s - 3ms/step - loss: 6658.3433 - mae: 58.6267 - val_loss: 6185.3462 - val_mae: 55.7146 - learning_rate: 0.0224
Epoch 69/100
46/46 - 0s - 4ms/step - loss: 6658.4072 - mae: 58.6345 - val_loss: 6185.3911 - val_mae: 55.7195 - learning_rate: 0.0251
Epoch 70/100
46/46 - 0s - 4ms/step - loss: 6658.4888 - mae: 58.6413 - val_loss: 6185.4316 - val_mae: 55.7238 - learning_rate: 0.0282
Epoch 71/100
46/46 - 0s - 4ms/step - loss: 6658.5850 - mae: 58.6471 - val_loss: 6185.4673 - val_mae: 55.7273 - learning_rate: 0.0316
Epoch 72/100
46/46 - 0s - 4ms/step - loss: 6658.6953 - mae: 58.6519 - val_loss: 6185.4985 - val_mae: 55.7301 - learning_rate: 0.0355
Epoch 73/100
46/46 - 0s - 4ms/step - loss: 6658.8218 - mae: 58.6558 - val_loss: 6185.5229 - val_mae: 55.7323 - learning_rate: 0.0398
Epoch 74/100
46/46 - 0s - 4ms/step - loss: 6658.9609 - mae: 58.6588 - val_loss: 6185.5415 - val_mae: 55.7339 - learning_rate: 0.0447
Epoch 75/100
46/46 - 0s - 4ms/step - loss: 6659.1143 - mae: 58.6609 - val_loss: 6185.5542 - val_mae: 55.7350 - learning_rate: 0.0501
Epoch 76/100
46/46 - 0s - 4ms/step - loss: 6659.2822 - mae: 58.6624 - val_loss: 6185.5615 - val_mae: 55.7356 - learning_rate: 0.0562
Epoch 77/100
46/46 - 0s - 4ms/step - loss: 6659.4629 - mae: 58.6633 - val_loss: 6185.5654 - val_mae: 55.7359 - learning_rate: 0.0631
Epoch 78/100
46/46 - 0s - 3ms/step - loss: 6659.6577 - mae: 58.6638 - val_loss: 6185.5649 - val_mae: 55.7359 - learning_rate: 0.0708
Epoch 79/100
46/46 - 0s - 4ms/step - loss: 6659.8667 - mae: 58.6639 - val_loss: 6185.5635 - val_mae: 55.7357 - learning_rate: 0.0794
Epoch 80/100
46/46 - 0s - 4ms/step - loss: 6660.0854 - mae: 58.6638 - val_loss: 6185.5591 - val_mae: 55.7353 - learning_rate: 0.0891
Epoch 81/100
46/46 - 0s - 4ms/step - loss: 6660.3091 - mae: 58.6634 - val_loss: 6185.5522 - val_mae: 55.7348 - learning_rate: 0.1000
Epoch 82/100
46/46 - 0s - 4ms/step - loss: 6660.5322 - mae: 58.6629 - val_loss: 6185.5454 - val_mae: 55.7342 - learning_rate: 0.1122
Epoch 83/100
46/46 - 0s - 4ms/step - loss: 6660.7471 - mae: 58.6622 - val_loss: 6185.5361 - val_mae: 55.7334 - learning_rate: 0.1259
Epoch 84/100
46/46 - 0s - 4ms/step - loss: 6660.9487 - mae: 58.6614 - val_loss: 6185.5269 - val_mae: 55.7326 - learning_rate: 0.1413
Epoch 85/100
46/46 - 0s - 4ms/step - loss: 6661.1230 - mae: 58.6606 - val_loss: 6185.5166 - val_mae: 55.7317 - learning_rate: 0.1585
Epoch 86/100
46/46 - 0s - 3ms/step - loss: 6661.2661 - mae: 58.6598 - val_loss: 6185.5088 - val_mae: 55.7310 - learning_rate: 0.1778
Epoch 87/100
46/46 - 0s - 4ms/step - loss: 6661.3799 - mae: 58.6591 - val_loss: 6185.5044 - val_mae: 55.7306 - learning_rate: 0.1995
Epoch 88/100
46/46 - 0s - 4ms/step - loss: 6661.4756 - mae: 58.6590 - val_loss: 6185.5068 - val_mae: 55.7308 - learning_rate: 0.2239
Epoch 89/100
46/46 - 0s - 4ms/step - loss: 6661.5698 - mae: 58.6596 - val_loss: 6185.5181 - val_mae: 55.7319 - learning_rate: 0.2512
Epoch 90/100
46/46 - 0s - 4ms/step - loss: 6661.6919 - mae: 58.6612 - val_loss: 6185.5435 - val_mae: 55.7341 - learning_rate: 0.2818
Epoch 91/100
46/46 - 0s - 3ms/step - loss: 6661.8906 - mae: 58.6640 - val_loss: 6185.5825 - val_mae: 55.7373 - learning_rate: 0.3162
Epoch 92/100
46/46 - 0s - 4ms/step - loss: 6662.2090 - mae: 58.6676 - val_loss: 6185.6309 - val_mae: 55.7411 - learning_rate: 0.3548
Epoch 93/100
46/46 - 0s - 4ms/step - loss: 6662.6807 - mae: 58.6713 - val_loss: 6185.6807 - val_mae: 55.7449 - learning_rate: 0.3981
Epoch 94/100
46/46 - 0s - 4ms/step - loss: 6663.3130 - mae: 58.6743 - val_loss: 6185.7314 - val_mae: 55.7485 - learning_rate: 0.4467
Epoch 95/100
46/46 - 0s - 4ms/step - loss: 6664.1064 - mae: 58.6765 - val_loss: 6185.7783 - val_mae: 55.7518 - learning_rate: 0.5012
Epoch 96/100
46/46 - 0s - 4ms/step - loss: 6665.0068 - mae: 58.6781 - val_loss: 6185.8408 - val_mae: 55.7559 - learning_rate: 0.5623
Epoch 97/100
46/46 - 0s - 4ms/step - loss: 6666.0430 - mae: 58.6848 - val_loss: 6185.9263 - val_mae: 55.7614 - learning_rate: 0.6310
Epoch 98/100
46/46 - 0s - 4ms/step - loss: 6667.2344 - mae: 58.7013 - val_loss: 6186.0479 - val_mae: 55.7686 - learning_rate: 0.7079
Epoch 99/100
46/46 - 0s - 4ms/step - loss: 6668.6074 - mae: 58.7239 - val_loss: 6186.2227 - val_mae: 55.7781 - learning_rate: 0.7943
Epoch 100/100
46/46 - 0s - 4ms/step - loss: 6670.1899 - mae: 58.7511 - val_loss: 6186.4790 - val_mae: 55.7909 - learning_rate: 0.8913

Let's have a look at the graph of the learning rate scheduler. We'll compare it by the loss value of the trained model.

In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate

lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
No description has been provided for this image

I've actually ran this exact code from above about 10 times, and every result presents something very different. Out of all the results, it seems to either settle on 10^-3, or 10^1. So let's see how the results look between the two.

MODEL 7 - 0.001 Learning Rate for ADAM¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_7 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelwaist_7.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

modelwaist_7.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd3d4fcad0>

MODEL 8 - 0.1 Learning Rate for ADAM¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_8 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelwaist_8.compile(optimizer=Adam(learning_rate=0.1),
                loss='mse',
                metrics=['mae'])

modelwaist_8.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd402921d0>
In [ ]:
show_scores(modelwaist_7, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_8, ywaist_test, Xwaist_test, 'Waist Evaluation')

WARNING:tensorflow:5 out of the last 13 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x000002BD4A2E20C0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step
WARNING:tensorflow:5 out of the last 13 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x000002BD4A2E0680> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 19ms/step
Out[ ]:
(('Waist Evaluation',
  {'Test MAPE (%)': np.float64(97.92764344645435),
   'Test MSE': 6315.116589657433,
   'Test R^2': -0.021016330090229518}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(99.88777454403663),
   'Test MSE': 6185.132946333593,
   'Test R^2': -8.158722311080169e-07}))
In [ ]:
ywaist_pred1 = modelwaist_7.predict(Xwaist_test)
ywaist_pred2 = modelwaist_8.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred1, label='Weight prediction 0.001 lr')
ax.plot(ywaist_pred2, label='Waist prediction 0.1 lr')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
No description has been provided for this image

Alright, seems like 0.001 is better than 0.1, as the latter learning rate is way too strong, that it misses the many nuances of the data.

Let's compare the learning rate with the latest model of waist.

In [ ]:
ywaist_pred = modelwaist_6.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
No description has been provided for this image

Definitely seems to perform a hell of a lot better with the learning rate of 0.001!

But now that got me to think, should we also see how well other optimizers work on the dataset, rather than relying on Adam? My bias towards Adam, has been due to how it outperforms SGD on practically every data I've worked with. But we need to rule out all possibilities first.

ChatGPT has also suggested to use RMSprop, so we will use this in conjunction with SGD, and also try find their optimal learning rate.

In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)

sgd_model = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

sgd_model.compile(optimizer='sgd',
                loss='mse',
                metrics=['mae'])

# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase

find_lr_history = sgd_model.fit(Xwaist_train, 
                                   ywaist_train, 
                                   epochs=100, 
                                   verbose=0,
                                   validation_data=(Xwaist_test,ywaist_test),
                                   callbacks=[lr_scheduler])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate

lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
No description has been provided for this image
In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)

rms_model = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

rms_model.compile(optimizer='rmsprop',
                loss='mse',
                metrics=['mae'])

# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase

find_lr_history = rms_model.fit(Xwaist_train, 
                                   ywaist_train, 
                                   epochs=100, 
                                   verbose=0,
                                   validation_data=(Xwaist_test,ywaist_test),
                                   callbacks=[lr_scheduler])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate

lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
No description has been provided for this image

Hmmm both SGD and RMSprop don't seem to have optimal learning rates. So we'll settle with the default and see how it performs on the data.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_9 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelwaist_9.compile(optimizer='sgd',
                loss='mse',
                metrics=['mae'])

modelwaist_9.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd5c153610>
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_10 = keras.Sequential([
    layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
    layers.Dense(32, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(256, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])

modelwaist_10.compile(optimizer='rmsprop',
                loss='mse',
                metrics=['mae'])

modelwaist_10.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd5c7517d0>

We will visually compare the three different optimizers using graphs, and see which performs best visually.

In [ ]:
# display the three different optimizers together
sgd_pred = modelwaist_9.predict(Xwaist_test)
rms_pred = modelwaist_10.predict(Xwaist_test)
adam_pred = modelwaist_7.predict(Xwaist_test)

fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='SGD waist prediction', xlabel='Date')
ax[0].plot(sgd_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();

ax[1].set(title='RMSprop waist prediction', xlabel='Date')
ax[1].plot(rms_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();

ax[2].set(title='Adam waist prediction', xlabel='Date')
ax[2].plot(adam_pred, label='Waist prediction')
ax[2].plot(ywaist_test, label='True waist')
ax[2].legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
No description has been provided for this image

Wow... SGD just flat out doesn't exist lmao.

With the comparison, Adam (0.001) and RMSprop are actually pretty comparable with each other. Despite RMSprop correctly guessing a positive difference, compared to the negative trough of Adam (0.001), I think I'll stick to Adam, due to the slight variability it has over RMSprop.

Other improvements?¶

So far, the improvements aren't enough to really learn the patterns of the data. They still hover at 0 a lot. Again, I asked ChatGPT for some guidance, to see what else I can add.

Recurrent Neural Networks (RNNs): Examples such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit), where they capture sequential dependencies. Aka, they look at rolling past data, e.g. the past 7 days, rather than looking at just one day to improve pattern recognition. They are great on time-series data, and can recognize longer-term dependencies/trends.

Other Functions:

  • LeakyReLU: Fixes ReLU, by helping it avoid dying neuron issues
  • Tanh: Helps if our training data hovers around 0
  • Dropout: Prevents overfitting through disabling a few neurons at random, to prevent over-reliance on certain neurons
  • Batch Normalization: Stabilizes training, improves gradient flow, and prevents internal covariate shift

MODEL 11 - Prepearing for LSTM¶

To perform LSTM on our data, we'lll need to reshape the data to a 3D array. Where we have (X_data, y_data, window_size) > windows_size is the value for how far the model will view the data from the past, e.g. 7 days in the past.

This helps the model catch short trends over a week, reducing short-term noises.

In [ ]:
# define a function to create a time-series sequence
def create_sequences(X,y,window_size):
    Xs, ys = [], []
    for i in range(len(X) - window_size):
        Xs.append(X[i:i+window_size])
        ys.append(y[i+window_size])
    return np.array(Xs), np.array(ys)

window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)

ReLU activation replaced with tanh¶

With the LSTM layers, the activation can't use relu, as they can't squish values between -1 and 1, which can lead to exploding activations and lack of control. So in place is tanh suggested by ChatGPT. It also stated, that ReLU would rather be ideal if it comes after the LSTM layers.

for this model, we'll try LSTM on the first half of increasing layers, and then decrease in ReLU.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_11 = keras.Sequential([
    layers.LSTM(16, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(32, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=True),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(256, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_11.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

modelwaist_11.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=0)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd7b0317d0>
In [ ]:
ywaist_pred = modelwaist_11.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 166ms/step
No description has been provided for this image

Wow that was very disappointing...

Maybe we need to change up the neurons and layers of LSTM? ChatGPT has made a model example, where the neuron layer is as follows > 64, 128, 64 for LSTM. So let's try that.

MODEL 12 - Reorganize LSTM layer/neurons¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_12 = keras.Sequential([
    layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_12.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

modelwaist_12.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=0)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd871eed10>
In [ ]:
ywaist_pred = modelwaist_12.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 108ms/step
No description has been provided for this image

Again there's nothing... let's have a visual of the ML model's verbose detail, and see if we can troubleshoot it.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_12 = keras.Sequential([
    layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_12.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist = modelwaist_12.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=2)

Epoch 1/20

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)

46/46 - 6s - 121ms/step - loss: 6670.5825 - mae: 58.5628
Epoch 2/20
46/46 - 0s - 9ms/step - loss: 6670.1753 - mae: 58.5577
Epoch 3/20
46/46 - 0s - 9ms/step - loss: 6670.1016 - mae: 58.5735
Epoch 4/20
46/46 - 0s - 9ms/step - loss: 6670.0640 - mae: 58.5756
Epoch 5/20
46/46 - 0s - 9ms/step - loss: 6669.9160 - mae: 58.5841
Epoch 6/20
46/46 - 0s - 9ms/step - loss: 6669.8730 - mae: 58.5923
Epoch 7/20
46/46 - 0s - 9ms/step - loss: 6669.8228 - mae: 58.5991
Epoch 8/20
46/46 - 0s - 9ms/step - loss: 6669.7583 - mae: 58.6074
Epoch 9/20
46/46 - 0s - 10ms/step - loss: 6669.7158 - mae: 58.6146
Epoch 10/20
46/46 - 0s - 9ms/step - loss: 6669.6553 - mae: 58.6228
Epoch 11/20
46/46 - 0s - 9ms/step - loss: 6669.6519 - mae: 58.6279
Epoch 12/20
46/46 - 0s - 9ms/step - loss: 6669.5977 - mae: 58.6323
Epoch 13/20
46/46 - 0s - 9ms/step - loss: 6669.5645 - mae: 58.6386
Epoch 14/20
46/46 - 0s - 10ms/step - loss: 6669.5811 - mae: 58.6400
Epoch 15/20
46/46 - 0s - 9ms/step - loss: 6669.4985 - mae: 58.6443
Epoch 16/20
46/46 - 0s - 10ms/step - loss: 6669.4346 - mae: 58.6494
Epoch 17/20
46/46 - 0s - 9ms/step - loss: 6669.4727 - mae: 58.6517
Epoch 18/20
46/46 - 0s - 9ms/step - loss: 6669.4189 - mae: 58.6543
Epoch 19/20
46/46 - 0s - 9ms/step - loss: 6669.3823 - mae: 58.6571
Epoch 20/20
46/46 - 0s - 9ms/step - loss: 6669.3110 - mae: 58.6647
In [ ]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

So it seems like Loss is going down, but very slowly. Let's try and train it on 100 epochs to see if that makes a difference.

MODEL 13 - 100 Epochs for LSTM MODEL 12¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_13 = keras.Sequential([
    layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_13.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_13 = modelwaist_13.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_13.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

It's improving sure, but it's very marginal. Not going to be efficient if we train this for 1000 epochs, as our resources and time are limited. Let's see how our MODEL 11 with 5 LSTM models do.

MODEL 14 - MODEL 11 has 100 epochs¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_14 = keras.Sequential([
    layers.LSTM(16, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(32, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=True),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(256, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_14.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_14 = modelwaist_14.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
In [ ]:
plt.plot(hist_14.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Performance is pretty much stalled for the majority of the learning progress... No real progress made until the last 10 epochs...

But what if, the time sequence is too long? We've set it to 7 days, but what if after 7 days of weight and waist difference, the positive gain, is cancelled out with the negative loss? So let's set the time sequence to 3 days, and see how it affects the results with loss.

MODEL 15 - Set Time Sequence for LSTM to 3 days¶

In [ ]:
window_size=3 # changing from 7 to 3
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_15 = keras.Sequential([
    layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_15.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_15 = modelwaist_15.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
In [ ]:
plt.plot(hist_15.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Doesn't seem to help with accuracy. Let's try a time series of 14 days instead

MODEL 16 - Time Series extended to 14 instead of 3¶

In [ ]:
window_size=14 # changing from 3 to 14
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_16 = keras.Sequential([
    layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.LSTM(128, activation='tanh', return_sequences=True),
    layers.LSTM(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_16.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_16 = modelwaist_16.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_16.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Seems 7 days is the sweet spot.

MODEL 17 - Switch LSTM for GRU¶

ChatGPT suggested that if LSTM is not performing as expected, that we can try using GRU layers, which does something similar to LSTM, except it is more efficient and works well with smaller datasets like ours.

In [ ]:
window_size=7 # window_size back to 7
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_17 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_17.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_17 = modelwaist_17.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_17.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Looks like we're finally getting somewhere!

MODEL 18 - Increase GRU layers/neurons¶

Now I want to see what happens if I try increasing the GRU layers, where there's 256 neurons on the biggest GRU layer

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_18 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(256, activation='tanh', return_sequences=True),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='relu'),
    layers.Dense(64, activation='relu'),
    layers.Dense(32, activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(8, activation='relu'),
    layers.Dense(1, activation='linear')
])


modelwaist_18.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_18 = modelwaist_18.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_18.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

So we double the time to train the model, to get essentially no improvement :(

MODEL 19 - Change ReLU to Tanh¶

ChatGPT has stated that Tanh might help with data around zero, so we'll replace ReLU with Tanh and see how it performs.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_19 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='tanh'),
    layers.Dense(64, activation='tanh'),
    layers.Dense(32, activation='tanh'),
    layers.Dense(16, activation='tanh'),
    layers.Dense(8, activation='tanh'),
    layers.Dense(1, activation='linear')
])


modelwaist_19.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_19 = modelwaist_19.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_19.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Doesn't work

MODEL 20 - Try LeakyReLU¶

Let's try using LeakyReLU in-place of Tanh (that aren't a part of GRU)

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_20 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_20.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_20 = modelwaist_20.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_20.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Best performance so far!

In [ ]:
ywaist_pred = modelwaist_20.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 140ms/step
No description has been provided for this image

The graph so far reflects the risk taking! But we need it to keep doing it for all areas of the data

MODEL 21 - Implement Batch Normalization and Dropout¶

ChatGPT suggests that I try adding Batch Normalization and Dropout into my layers.

Normalization is to help reduce internal covariate shift, making training more stable. They normalize activations by subtracting the mean, and dividing by standard deviation. This is often added after a Dense/Convolutional layer, but before activation functions.

Dropout is to help reduce overfitting by preventing over-reliance to certain neurons. This is done by randomly dropping some neurons to the value of 0, forcing the model to learn patterns without such neuron. It's added after a Dense or LSTM/GRU layer.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_21 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.BatchNormalization(),
    layers.Dropout(0.2),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.BatchNormalization(),
    layers.Dropout(0.2),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),

    layers.Dense(128),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(16),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(8),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(1, activation='linear')
])


modelwaist_21.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_21 = modelwaist_21.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_21.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Didn't work as hoped. Maybe I added it to too many layers?

MODEL 22 - Reduce Batch Normzlization and Dropout layers¶

This time, I'm going to only add Batch Normalization and Dropout layers to 2 GRU layers. The first GRU layer will not have either of these layers.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_22 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.BatchNormalization(),
    layers.Dropout(0.2),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),

    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),

    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_22.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_22 = modelwaist_22.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_22.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

The model seems to have a hard time improving with Batch Normalization and Dropout

Model 23 - Use Batch Normalization and Dropout on LeakyReLU¶

Let's try using the Normalization and Dropout layer on the LeakyReLU layers only

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_23 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(16),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(8),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(1, activation='linear')
])


modelwaist_23.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_23 = modelwaist_23.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_23.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Seems like I'll have to retire Batch Normalization and Dropout from the layers in search of better options.

MODEL 24 - Intermittent switching between LeakyReLU and tanh¶

Since tanh is supposedly good at working with data around 0, I figure that swapping between LeakyReLU layers, and tanh layers would be interesting, and I hope that both their specialty will compliment each other.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_24 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64, activation='tanh'),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16, activation='tanh'),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_24.compile(optimizer=Adam(learning_rate=0.001),
                loss='mse',
                metrics=['mae'])

hist_24 = modelwaist_24.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_24.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Okay, maybe not then :(

It's back to the drawing board for me to search for solutions to better this ML model. So I've once again asked for guidance with ChatGPT, especially with the model not being risk-taking, and showed the best resulting ML model so far. It has given me suggestions to help it take risks and be less underfitting, so I'll work through each one and see what results we get.

MODEL 25 - Change MSE to Huber Loss, but also finding the ideal delta variable¶

Huber loss is less sensitive to extreme errors, which allows the model to take risks. One extra thing is it has this variable named delta. It acts like a scaled slider between mse and mae. The smaller the delta is, the more it'll become mse. Vice versa for larger delta.

Small delta > less risk taking, Large delta > more risk taking

Problem is, how do we find ideal Delta? ChatGPT has given me a line of code that suggests the ideal Delta. By finding the difference of prediction to truth, getting it's standard deviation, and multiply it by 1.5

In [ ]:
import numpy as np

# Compute residuals
residuals = np.abs(ywaist_sequence_test - ywaist_pred)  # Get difference of preds and truth using .abs

# Compute standard deviation
delta = 1.5 * np.std(residuals)

print(f"Ideal Delta: {delta}")

Ideal Delta: 93.23351755665132

Now let's try it on the model. Because we've changed our loss function, and is going to change based on the delta, we'll compare using the values it predicts.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_25 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_25.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_25 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_25.predict(expandX_test)

fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();

ax[1].set(title='Huber Delta=93.2 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();

ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_25.history['loss'])

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be2f5f2750>]
No description has been provided for this image

Hmmm no extreme risk taking, might need to increase delta

MODEL 26 - Delta set to 1000¶

Might as well try the most extreme of extreme. If a Delta of 93.2 is not doing much, we might as well try an extreme value, and see how the ML model reacts to it.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_26 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_26.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=1000),
                metrics=['mae'])

hist_26 = modelwaist_26.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_26.predict(expandX_test)

fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();

ax[1].set(title='Huber Delta=1000 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();

ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_26.history['loss'])

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be4536bb50>]
No description has been provided for this image

Uh oh, seems to do way worse, despite the higher the value, the more risk taking it does!

MODEL 27 - Delta GridSearchCV¶

One other thing ChatGPT suggested, is by using GridSearchCV, with multiple different delta values. And then having it return the gest delta value, based on the presets we've given to the model.

I've set the GridSearch upon the best model so far, model_20 and will see what happens.

In [ ]:
from tensorflow.keras.losses import Huber
from tensorflow.keras.optimizers import Adam

deltas = [1, 5, 10, 20, 50]
best_delta = None
best_val_loss = float("inf")

for delta in deltas:
    modelwaist_20.compile(optimizer=Adam(learning_rate=0.001), loss=Huber(delta=delta), metrics=['mae'])
    history = modelwaist_20.fit(expandX_train, ywaist_sequence_train, validation_data=(expandX_test, ywaist_sequence_test), epochs=50, verbose=0)
    
    val_loss = min(history.history['val_loss'])  # Get best validation loss
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_delta = delta

print(f"Optimal Delta: {best_delta}")

Optimal Delta: 1

Surprising it gave 1 as the best delta value, despite smaller value mean less risk taking.

Regardless, let's try it on model

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_27 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])


modelwaist_27.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=1000),
                metrics=['mae'])

hist_27 = modelwaist_27.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_27.predict(expandX_test)

fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();

ax[1].set(title='Huber Delta=1 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();

ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_27.history['loss'])

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be4abbc610>]
No description has been provided for this image

Aah I see why that's best performing, because without taking risks and staying at 0, you are closest to all predictions, over risking wrong predictions!

MODEL 28 - Reduce Adam Learning Rate Overtime¶

ChatGPT theorized that, the learning rate for Adam, is making the model stuck in doing small changes, and settling too ealy near zero. Reducing learning rate overtime, helps it fine-tune its predictions, rather than getting stuck on bad local minimums.

We'll pull in MODEL_25 and try our learning rate reducing experiment on it.

In [ ]:
from tensorflow.keras.callbacks import ReduceLROnPlateau

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-5)     # Do not reduce below this

# Include in model training
hist_28 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])

Using the MODEL_25, means we can compare loss meaningfully. Based on the graph, we need something lower than 2757 loss to show that there's improvement.

In [ ]:
plt.plot(hist_28.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Results! What if we made the minimum learning rate, even smaller in value?

MODEL_29 - min_lr becomes 1e-10, and factor becomes 0.25¶

Again, trying some extreme values to see what happens

In [ ]:
from tensorflow.keras.callbacks import ReduceLROnPlateau

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.25,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

# Include in model training
hist_29 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_29.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Hmm it started with a much lower loss compared to MODEL_28, but just doesn't improve anymore.

In [ ]:
ywaist_pred = modelwaist_25.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
No description has been provided for this image

Seems we're encountering a pattern, where the model is only predicting at the same are of the graph... However, it is making the kind of risk taking prediction that I've been looking for!

Let's try one more thing before we dive in deeper.

MODEL_30 - Use Swish over LeakyReLU / decreased min_lr¶

Last thing ChatGPT suggested, is swapping LeakyReLU for swish. What it does, it allows small negative values, whereas ReLU outputs zero on negative inputs. This can block gradient flow. Swish smoothly transitions from negative to positive values, works better in deeper models with gradient flow improvement, and has a value named x in the equation, to allow the learning of when to supress or amplify values.

I've decided to further decrease the minimum learning rate, to see what happens

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_30 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128, activation='swish'),
    layers.Dense(64, activation='swish'),
    layers.Dense(32, activation='swish'),
    layers.Dense(16, activation='swish'),
    layers.Dense(8, activation='swish'),

    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.25,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-20)     # Do not reduce below this

modelwaist_30.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_30 = modelwaist_30.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
In [ ]:
plt.plot(hist_30.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [ ]:
ywaist_pred = modelwaist_30.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 152ms/step
No description has been provided for this image

Not too impressive, but it seems to have the same issue with most if not all previous model. All predictions is fitting within the area between index 10 to index 40 on the x-axis!

MODEL_31 - reintroduce Normalizaton and Dropout¶

Again ChatGPT suggest we solve the issue with Batch Normalization and Dropout. This time, I've reduced the number of these layers, and only added them, after the 3 GRU layers.

Instead of factor=0.25 in Learning Rate Scheduler, I've set it back to 0.5, as it did show promising results of continually decreasing loss values.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_31 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_31.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_31 = modelwaist_31.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_31.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Getting some slightly better results!

MODEL_32 - Add more Batch Normalization and Dropout¶

With further experimentation, I'm just going to add 2 more Batch Normalization and Dropout in the layer as I see fit, and see the resulting outcome.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_32 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.BatchNormalization(),
    layers.Dropout(0.2),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.BatchNormalization(),
    layers.Dropout(0.2),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_32.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_32 = modelwaist_32.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_32.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Not good results

MODEL_33 - Epochs increase to 1,000¶

One big hail mary, is to increase the epoch. I notice how a lot of the graphs show how Loss seems to further decrease at a constant rate the more epochs there is. I'm hesitant in increasing epochs, as it's one of the easiest way to increase training time. However I'm close to exhausting all my options, so I might as well try this one time and see the results.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_33 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_33.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_33 = modelwaist_33.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_33.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

We can see how loss continuoually decreases linearly, and reaches a point where it stops improving. Likely because Learning Rate of Adam becomes too small.

In [ ]:
ywaist_pred = modelwaist_33.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 127ms/step
No description has been provided for this image

Finally! We're seeing predictions outside the index 10 to index 40 area!

The likely reason why there are 0 predictions, could be because the ML model didn't have enough time to train and familiarize in all aspects of the data. It probably only recognize the most prominent parts of the data. Aka, the difference value that's most extreme (around index 10 to index 40). Only has it been trained even more, can it effectively predice smaller difference value.

MODEL_34 - Increase factor from 0.5 to 0.8¶

I'll increase factor. Due to our loss graph, we are decreasing the Learning Rate too fast, that it flattens out and doesn't learn anything meaningful. So increasing it allows more time for higher Learning Rate epochs to make a dent on prediction.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_34 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_34.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_34 = modelwaist_34.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_34.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

We're getting even better results!

This time, it's flattening around epoch 400

In [ ]:
ywaist_pred = modelwaist_34.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 129ms/step
No description has been provided for this image

Again, the more the model learns, the more likely the model starts to make predictions!

MODEL_35 - factor 0.8 to 0.92¶

One thing I wanted to figure out the cutoff for where the loss starts to flatten out. We know Learning Rate starts at 0.001, and every 10 epochs, it reduces by 80%. But we want to find the Learning Rate at Epoch 400, where Loss flattens.

This comes out to the following equation:

$\\ \text{lr}_n = \text{lr}_0 \times (0.8)^{\frac{n}{10}} $

$\\ \text{lr}_{400} = 0.001 \times (0.8)^{\frac{400}{10}} = 0.001 \times (0.8)^{40} \approx 1.33 \times 10^{-7} $

1.33 * 10^-7 is the approximate Learning Rate where learning flattens. Let's extrapolate it, and set that at Epoch 1000, rather than on Epoch 400.

$\\ \ 1.33 \times 10^{-7} = 0.001 \times (r)^{\frac{1000}{10}} -> r = \left(\frac{1.33 \times 10^{-7}}{\text{0.001}}\right)^{\frac{10}{1000}} $

Based on the equation and rearrangement, the factor is around 0.9146, or reducing by 8.54% every 10 epochs. Based on that, I'll try setting the factor at 0.92, a little higher than suggested so reduceis slower.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_35 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.92,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_35.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_35 = modelwaist_35.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[419], line 45
     36 reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
     37                               factor=0.92,      # Reduce LR by half when triggered
     38                               patience=10,     # Wait 10 epochs before reducing
     39                               min_lr=1e-10)     # Do not reduce below this
     41 modelwaist_35.compile(optimizer=Adam(learning_rate=0.001),
     42                 loss=Huber(delta=93.2),
     43                 metrics=['mae'])
---> 45 hist_35 = modelwaist_35.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py:117, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    115 filtered_tb = None
    116 try:
--> 117     return fn(*args, **kwargs)
    118 except Exception as e:
    119     filtered_tb = _process_traceback_frames(e.__traceback__)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py:371, in TensorFlowTrainer.fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq)
    369 for step, iterator in epoch_iterator:
    370     callbacks.on_train_batch_begin(step)
--> 371     logs = self.train_function(iterator)
    372     callbacks.on_train_batch_end(step, logs)
    373     if self.stop_training:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py:219, in TensorFlowTrainer._make_function.<locals>.function(iterator)
    215 def function(iterator):
    216     if isinstance(
    217         iterator, (tf.data.Iterator, tf.distribute.DistributedIterator)
    218     ):
--> 219         opt_outputs = multi_step_on_iterator(iterator)
    220         if not opt_outputs.has_value():
    221             raise StopIteration

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\util\traceback_utils.py:150, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    148 filtered_tb = None
    149 try:
--> 150   return fn(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py:833, in Function.__call__(self, *args, **kwds)
    830 compiler = "xla" if self._jit_compile else "nonXla"
    832 with OptionalXlaContext(self._jit_compile):
--> 833   result = self._call(*args, **kwds)
    835 new_tracing_count = self.experimental_get_tracing_count()
    836 without_tracing = (tracing_count == new_tracing_count)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py:878, in Function._call(self, *args, **kwds)
    875 self._lock.release()
    876 # In this case we have not created variables on the first call. So we can
    877 # run the first trace but we should fail if variables are created.
--> 878 results = tracing_compilation.call_function(
    879     args, kwds, self._variable_creation_config
    880 )
    881 if self._created_variables:
    882   raise ValueError("Creating variables on a non-first call to a function"
    883                    " decorated with tf.function.")

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\tracing_compilation.py:139, in call_function(args, kwargs, tracing_options)
    137 bound_args = function.function_type.bind(*args, **kwargs)
    138 flat_inputs = function.function_type.unpack_inputs(bound_args)
--> 139 return function._call_flat(  # pylint: disable=protected-access
    140     flat_inputs, captured_inputs=function.captured_inputs
    141 )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\concrete_function.py:1322, in ConcreteFunction._call_flat(self, tensor_inputs, captured_inputs)
   1318 possible_gradient_type = gradients_util.PossibleTapeGradientTypes(args)
   1319 if (possible_gradient_type == gradients_util.POSSIBLE_GRADIENT_TYPES_NONE
   1320     and executing_eagerly):
   1321   # No tape is watching; skip to running the function.
-> 1322   return self._inference_function.call_preflattened(args)
   1323 forward_backward = self._select_forward_and_backward_functions(
   1324     args,
   1325     possible_gradient_type,
   1326     executing_eagerly)
   1327 forward_function, args_with_tangents = forward_backward.forward()

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\atomic_function.py:216, in AtomicFunction.call_preflattened(self, args)
    214 def call_preflattened(self, args: Sequence[core.Tensor]) -> Any:
    215   """Calls with flattened tensor inputs and returns the structured output."""
--> 216   flat_outputs = self.call_flat(*args)
    217   return self.function_type.pack_output(flat_outputs)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\atomic_function.py:251, in AtomicFunction.call_flat(self, *args)
    249 with record.stop_recording():
    250   if self._bound_context.executing_eagerly():
--> 251     outputs = self._bound_context.call_function(
    252         self.name,
    253         list(args),
    254         len(self.function_type.flat_outputs),
    255     )
    256   else:
    257     outputs = make_call_op_in_graph(
    258         self,
    259         list(args),
    260         self._bound_context.function_call_options.as_attrs(),
    261     )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\context.py:1683, in Context.call_function(self, name, tensor_inputs, num_outputs)
   1681 cancellation_context = cancellation.context()
   1682 if cancellation_context is None:
-> 1683   outputs = execute.execute(
   1684       name.decode("utf-8"),
   1685       num_outputs=num_outputs,
   1686       inputs=tensor_inputs,
   1687       attrs=attrs,
   1688       ctx=self,
   1689   )
   1690 else:
   1691   outputs = execute.execute_with_cancellation(
   1692       name.decode("utf-8"),
   1693       num_outputs=num_outputs,
   (...)
   1697       cancellation_manager=cancellation_context,
   1698   )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     51 try:
     52   ctx.ensure_initialized()
---> 53   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     54                                       inputs, attrs, num_outputs)
     55 except core._NotOkStatusException as e:
     56   if name is not None:

KeyboardInterrupt:
In [ ]:
plt.plot(hist_35.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Not what I expected. Despite slower reduction, Loss didn't reduce as I thought. Maybe because once it reaches to a certain amount of epochs, it stops learning meaningful information?

MODEL_36 - Change Factor from 0.92 to 0.9¶

Lowering Factor just by a tiny bit, to see what the result is in Loss. See if my hypothesis of learning less meaning info after a certain amount of epochs.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_36 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.9,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_36.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_36 = modelwaist_36.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_36.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Seems like our hypothesis is somewhat correct. The range of epoch where it starts to flatten sits from 400 to 600.

What's good is the lower of Factor also improved Loss value.

So now we know to cut Epochs to 600, and save time in model training, and the range of optimal factor is between 0.8 to 0.9.

MODEL_37 - Set Epoch to 600, and factor to 0.85¶

Epoch to 600, based on previous training results, where improvement flattens around 400 to 600

Factor to 0.85, as we tried 0.8 and 0.9, assuming the ideal value is between these two numbers, based on the resulting Loss graphs.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_37 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False
    layers.BatchNormalization(),
    layers.Dropout(0.2),


    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(32),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(16),
    layers.LeakyReLU(alpha=0.1),
    layers.Dense(8),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.85,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_37.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_37 = modelwaist_37.fit(expandX_train, ywaist_sequence_train, epochs=600, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_37.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Further tests:¶

Now training each singular model is taking so much time, and takes up so much space through this notebook, I figured that I just run through many different options, and make a list of the changes I've made and what happened.

  • Factor - 0.7 > 0.6 > 0.75: There has no significant improvement when changing the factor

  • Delta - 150: I thought about changing the delta, and see if it could improve model. No improvement

  • Lr 0.01: I thought that, maybe the model is reaching a limit in the learning rate and its reaching too quickly, causing it to flatten. So I made the learning rate higher to make it reach that boundary slower. Didn't work

  • Patience 20: On the similar vein of Lr 0.01, where we slow down reaching the learning rate, that flattens training rate. No improvement

  • Epoch 200: I wanted to see what happens if epoch was cut down to a point before the flattening. Results showed, that the model still flattened earlier than the last 200 epochs

  • Return back to best model: I wanted to see if the loss is still the same with the same model. Surprisingly, no. It flattened out earlier, and did not improve at the same rate.

There's something wrong... Again returned to chatgpt for some answers. It states that the model is getting stuck at a local minimum, and plateauing at ranomd epochs. It can suggest that training dynamics are unstable.

So out of experimentation reasoning, I'll have chatgpt recreate the model with what it thinks will improve the flattening

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
from tensorflow.keras.callbacks import ReduceLROnPlateau

tf.random.set_seed(35)

expandX_train = Xwaist_sequence_train  # No np.expand_dims
expandX_test = Xwaist_sequence_test


# Learning rate schedule
initial_lr = 0.001
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_lr, decay_steps=200, decay_rate=0.96, staircase=True
)

# Create model
modelwaist_38 = keras.Sequential([
    layers.Bidirectional(layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2]))),
    layers.Bidirectional(layers.GRU(128, activation='tanh', return_sequences=True)),
    layers.Bidirectional(layers.GRU(64, activation='tanh', return_sequences=False)),  
    layers.Dropout(0.3),
    layers.BatchNormalization(),

    layers.Dense(128, activation='swish'),
    layers.Dense(64, activation='swish'),
    layers.Dense(32, activation='swish'),
    layers.Dense(16, activation='swish'),
    layers.Dense(8, activation='swish'),

    layers.Dense(1, activation='linear')
])

optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=20, restore_best_weights=True)

reduce_lr = ReduceLROnPlateau(monitor='loss',
                              factor=0.5,
                              patience=5,
                              min_lr=1e-6)

modelwaist_38.compile(optimizer=optimizer,
                loss=Huber(delta=50),  # Adjusted delta
                metrics=['mae'])

hist_38 = modelwaist_38.fit(expandX_train, ywaist_sequence_train, 
                            epochs=1000, 
                            verbose=0, 
                            callbacks=[reduce_lr, early_stop])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\callbacks\early_stopping.py:153: UserWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are: loss,mae,learning_rate
  current = self.get_monitor_value(logs)

Changes¶

  1. Change batch normalization and dropout order - The order of these two layers were not optimal to improvement, so they've been switched
  2. Overfitting or underfitting - To prevent that, we add early stopping when model stagnates or shows no improvements
  3. Trying different optimizers - Changing Adam to SGD, might work better
In [ ]:
plt.plot(hist_38.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [ ]:
ywaist_pred = modelwaist_38.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 2s 285ms/step
No description has been provided for this image

Didn't even work

CHAT GPT MODEL¶

I want to see what ChatGPT generates for a model, and see how different it is from my best performing model, and whether it would be better in prediction as well.

In [ ]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.losses import Huber

# Load or preprocess your dataset
# expandX_train, ywaist_sequence_train = ...
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)


# Define the model
gptmodel = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),  
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),

    layers.Dense(64, activation=None),
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),

    layers.Dense(32, activation=None),
    layers.BatchNormalization(),
    layers.ReLU(),

    layers.Dense(1)  # Output layer for regression
])

# Define optimizer with a fixed learning rate
optimizer = keras.optimizers.Adam(learning_rate=0.0005)

# Learning rate scheduler
reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.5, patience=50, min_lr=1e-6)

# Compile the model
gptmodel.compile(optimizer=optimizer, loss=Huber(delta=50), metrics=['mae'])

# Train the model
gpthistory = gptmodel.fit(expandX_train, ywaist_sequence_train, 
                    epochs=1000, verbose=1, callbacks=[reduce_lr])
In [ ]:
plt.plot(gpthistory.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Bingo! Finally the flattening loss has been solved. Let's see how well it performs on the test data as well.

In [ ]:
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)  # (None, 210)
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)  # (None, 210)

ywaist_pred = gptmodel.predict(expandX_test)

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
In [ ]:
ywaist_pred = gptmodel.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
No description has been provided for this image

Seems like it has finally done what I've been looking for this whole time!

The next step for me is to try and understand what made this model, so much better than my own fabricated one?

Difference between GPT Model to My Model¶

3D Layer Redundant: GPT has chosen to remove the 3 dimensional layer, and keep it as it was (2 dimensional). But why is that?

GRU Layer gone: The main purpose for 3D layer was for GRU. The 3rd dimension was used for sequential data, to keep a range of days before the current day. This helps GRU look at previous days data to make a prediction.

Batch Normalization After Dense Layer: GPT's strategy is putting Normalization layers after a Dense layer of neurons.

Dropout After Activations: GPT's strategy is putting the Dropout layers after an activation function, such as ReLU or Tanh. This is not done on the last activation, as right after is a 1 neuron Dense layer, which won't do any pattern recognition.

Activations changed: LeakyReLU has been substituted for ReLU.

Reduced Activation Layers: GPT has heavily reduced the layers in the model, keeping just 3 ReLU layers, from 128, to 64, 32 neurons.

Adam Learning Rate Hyperparameter at 0.0005: GPT chosen an even smaller learning rate.

Reduce LR Hyperparameter Change: Factor = 0.5, Patience = 50, min_lr = 1e-6. What's different from ours, is Learning Rate is reduced faster, at 50%, however it's reduced even slower, by every 50 epochs, while increasing the minimum Learning Rate from ^-10, to ^-6.

Huber Hyperparameter change: Delta is changed to 50, from 93.2

What point will improve most?¶

There's a good number of changes, but what would yield the most improvement to our current model? Based on what I've experimented so far, the ones that are based on hyperparameter change, will have minimal change in effect.

What would be most contentious, is the GRU, and Normalization/Dropout layer order. GRU essentially changes the entire training data to fit itself, and the way you organize Normalization and Dropout can significantly change the data that the other layers recieve. Plus, ChatGPT has srutinized how my Nomralization/Dropout layers are not placed optimally. The ReLU layers along with the simplicity might play a role as well.

Next step:¶

I'm going to now experiment with the GRU layers, and Normalization/Dropout layers with my current model, in the hopes that from what I've learnt with the GPTmodel, I can create an even more accurate model, and avoid loss flattening. If that doesn't work, then I'll look at minimizing layers as well.

MODEL_39 - Refit Normalization and Dropout¶

I'm going to copy the way ChatGPT organized and places its Batch Normalization and Dropout layers.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_39 = keras.Sequential([
    layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
    layers.GRU(128, activation='tanh', return_sequences=True),
    layers.GRU(64, activation='tanh', return_sequences=False),  # Last LSTM layer should have return_sequences=False

    layers.Dense(128),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(16),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(8),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_39.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_39 = modelwaist_39.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_39.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

It doesn't seem to make much of an improvement

MODEL_40 - Remove GRU layers¶

Remove the 3 GRU layers on the top. Since these layers use 3 dimensional training data to learn, we will also need to remove the expansion as well, and return it back to 2 dimensional for TensorFlow.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_40 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(16),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),
    
    layers.Dense(8),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_40.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_40 = modelwaist_40.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_40.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Seems like strong correlation that having RNN layers like GRU is what's gatekeeping the model from making any real progression/learning.

Definitely feel pretty shitty how much time was spent to perfect the model using GRU, but now we know to keep it on ReLu or something related.

MODEL_41 - Replace LeakyReLU with ReLU¶

Further copying ChatGPT and see if the change in activations will make a difference.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_41 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),

    layers.Dense(16),
    layers.BatchNormalization(),
    layers.ReLU(),
    layers.Dropout(0.2),
    
    layers.Dense(8),
    layers.BatchNormalization(),
    layers.ReLU(),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_41.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_41 = modelwaist_41.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_41.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Seems like LeakyReLu and ReLU are quite easily matched! Since ChatGPT stated that LeakyReLU is a bit better than ReLU, we'll stick to it.

MODEL_42 - Removed 16 and 8 neuron LeakyReLU layers¶

Copy the layer layout and removing 16 and 8 neuron layers. Having so many small neuron layers, might make it hard for the model to give varied predictions, as it's limited to a few options only.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_42 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_42.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_42 = modelwaist_42.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_42.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Yep that seem's to make it better a bit.

MODEL_43 - Reinsert one GRU layer¶

Just to be extra sure of my current conclusion, I want to try one GRU layer, and see if it really creates the previous issue we had, which is flattening in performance.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1)  # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)


# create model including relu and linear
modelwaist_43 = keras.Sequential([
    layers.GRU(256, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),

    layers.Dense(128), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.8,      # Reduce LR by half when triggered
                              patience=10,     # Wait 10 epochs before reducing
                              min_lr=1e-10)     # Do not reduce below this

modelwaist_43.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_43 = modelwaist_43.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_43.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Our conclusion is proven correct again.

MODEL_44 - Set reduce_lr to ChatGPT's model¶

Changed hyperparameter to match the ones suggested by ChatGPT's model

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_44 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=50,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_44.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=93.2),
                metrics=['mae'])

hist_44 = modelwaist_44.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_44.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Accuracy up! Seems the lowering of learning rate could do a little slowing down, to let the model learn on the most optimal learning rate for as long as possible.

MODEL_45 - learning_rate=0.0005, delta=50¶

Further copying ChatGPT's suggested model

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_45 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=50,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_45.compile(optimizer=Adam(learning_rate=0.0005),
                loss=Huber(delta=50),
                metrics=['mae'])

hist_45 = modelwaist_45.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_45.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

More progress made

MODE_46 - Patience 50>80, delta 50>25¶

Increased patience, to allow the reduction of learning rate to slow more. I'm hoping this will keep the model on the optimal learning rates for as long as possible, and prevent it from quickly flattening.

Delta was changed, because it showed improvement when it was lowered from 93.2 to 50. I wonder what 25 would do?

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_46 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1), 
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=80,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_46.compile(optimizer=Adam(learning_rate=0.0005),
                loss=Huber(delta=25),
                metrics=['mae'])

hist_46 = modelwaist_46.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_46.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

MODEL_47 - Change LeakyReLU to Tanh¶

ChatGPT had stated before that Tanh worked well on data that's close to 0. I was wondering whether GRU layers were holding it back performance-wise, so I want to try it out when replacing LeakyReLU

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model with tanh
modelwaist_47 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])


reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=80,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_47.compile(optimizer=Adam(learning_rate=0.0005),
                loss=Huber(delta=25),
                metrics=['mae'])

hist_47 = modelwaist_47.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_47.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Tanh works better than LeakyReLU, and so we'll use it

MODEL_48 - patience 80>100, delta 25>10, learning_rate 0.0005>0.0001¶

Our previous changes of patience (higher) delta (lower) and learning rate (lower), has proven fruitful in lowering loss, and we'll continue to push the limits of these values altogether

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model with tanh
modelwaist_48 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])


reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=100,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_48.compile(optimizer=Adam(learning_rate=0.0001),
                loss=Huber(delta=10),
                metrics=['mae'])

hist_48 = modelwaist_48.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_48.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Hold up... I just realized that Loss is minimizing from it's starting point...

I just remembered that changing delta changes the weight between mae and mse. I want to double check if that's the case by visualizing graph prediction.

In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)


ywaist_pred = modelwaist_48.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
No description has been provided for this image

So we need to find a way to compare model improvement, when I change up delta. What we can do, is calculate the ratio between starting loss, and ending loss after 1000 epochs. We'll make a rough estimate on what it starts and ends.

MODEL_44 : 2800 > 350 = 8

MODEL_45 : 2000 > 350 = 5.71

MODEL_46 : 1200 > 300 = 4

MODEL_47 : 1200 > 250 = 4.8

MODEL_48 : 550 > 300 = 1.833

Based on the new model comparison, MODEL_44 theoretically should be the best performing model. What we'll do is compare the three model with the highest ratio.

In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)

# display the three different optimizers together
model44_pred = modelwaist_44.predict(expandX_test)
model45_pred = modelwaist_45.predict(expandX_test)
model47_pred = modelwaist_47.predict(expandX_test)

fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='MODEL_44 prediction', xlabel='Date')
ax[0].plot(model44_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();

ax[1].set(title='MODEL_45 prediction', xlabel='Date')
ax[1].plot(model45_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();

ax[2].set(title='MODEL_47 prediction', xlabel='Date')
ax[2].plot(model47_pred, label='Waist prediction')
ax[2].plot(ywaist_test, label='True waist')
ax[2].legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 40ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
No description has been provided for this image

The hypothesis seems to match somewhat, so we'll continue improving MODEL_44

MODEL_49 - Delta = 150¶

Set Delta to 150. Seems like the lower the Delta is, the smaller the ratio is from start to end loss value

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_49 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=50,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_49.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=150),
                metrics=['mae'])

hist_49 = modelwaist_49.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_49.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)


ywaist_pred = modelwaist_49.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
No description has been provided for this image

Perfect! The higher the Delta is, the more the model takes risks to predict the Waist Difference!

MODEL_50 - Delta 150 > 250, LeakyReLU > Tanh, Patience 50 > 80¶

Push Delta higher and see if the model continues to push it's risk taking. LeakyReLU to Tanh as it proved successful in another model. Patience 50 saw too quick of a drop off in loss graph, and Patience 100 still had more opportunity to continue decreasing loss. Patience 80 is me trying to find a middle ground between the two.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_50 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=80,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_50.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=250),
                metrics=['mae'])

hist_50 = modelwaist_50.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_50.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Both graph look quite similar, so it's hard to tell which is better. I remember making a function to show scores of my models, so we'll use that

In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)

show_scores(modelwaist_49, ywaist_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, expandX_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[521], line 3
      1 expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
----> 3 show_scores(modelwaist_49, ywaist_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, expandX_test, 'Waist Evaluation')

Cell In[165], line 36, in show_scores(model, truth, test, title)
     32 preds = model.predict(test)
     34 # calculating scores
     35 scores = {
---> 36     "Test MAPE (%)":mean_absolute_percent_error(truth, preds),
     37     "Test MSE":mean_squared_error(truth, preds),
     38     "Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y
     39 }
     41 return title, scores

Cell In[165], line 12, in mean_absolute_percent_error(y_true, y_pred)
      9 y_pred = np.squeeze(y_pred)
     11 # Task1 > If true is 0, and pred is not 0, set to 100%
---> 12 percent_error[(y_true == 0) & (y_pred != 0)] = 100
     14 # Task2 > If true and pred is 0, set to 0%
     15 percent_error[(y_true == 0) & (y_pred == 0)] = 0

ValueError: operands could not be broadcast together with shapes (164,) (150,)

Seems like our prediction and test data shape are different from each other!

In [ ]:
print(expandX_test.shape[0], ywaist_test.shape[0])

Indeed it is when printing it's shape out. There must be something we did in the past that have altered the data shape, and it's likely the moving average/data window.

In [ ]:
# define a function to create a time-series sequence
def create_sequences(X,y,window_size):
    Xs, ys = [], []
    for i in range(len(X) - window_size):
        Xs.append(X[i:i+window_size])
        ys.append(y[i+window_size])
    return np.array(Xs), np.array(ys)

window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)

This is likely the original culprit, and we'll try retrain model_49 and model_50 with Xwaist_train/test.

Refit MODEL_49 and MODEL_50 on original training/testing data¶

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_50 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(Xwaist_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=80,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_50.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=250),
                metrics=['mae'])

hist_50 = modelwaist_50.fit(Xwaist_train, ywaist_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

# create model including relu and linear
modelwaist_49 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(Xwaist_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=50,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_49.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=150),
                metrics=['mae'])

hist_49 = modelwaist_49.fit(Xwaist_train, ywaist_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
show_scores(modelwaist_49, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, Xwaist_test, 'Waist Evaluation')

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
Cell In[558], line 1
----> 1 show_scores(modelwaist_49, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, Xwaist_test, 'Waist Evaluation')

Cell In[165], line 32, in show_scores(model, truth, test, title)
     29 def show_scores(model, truth, test, title):
     30     
     31     # model predicting values from training
---> 32     preds = model.predict(test)
     34     # calculating scores
     35     scores = {
     36         "Test MAPE (%)":mean_absolute_percent_error(truth, preds),
     37         "Test MSE":mean_squared_error(truth, preds),
     38         "Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y
     39     }

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    119     filtered_tb = _process_traceback_frames(e.__traceback__)
    120     # To get the full stack trace, call:
    121     # `keras.config.disable_traceback_filtering()`
--> 122     raise e.with_traceback(filtered_tb) from None
    123 finally:
    124     del filtered_tb

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     51 try:
     52   ctx.ensure_initialized()
---> 53   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     54                                       inputs, attrs, num_outputs)
     55 except core._NotOkStatusException as e:
     56   if name is not None:

InvalidArgumentError: Graph execution error:

Detected at node sequential_151_1/dense_935_1/BiasAdd defined at (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel_launcher.py", line 18, in <module>

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\traitlets\config\application.py", line 1075, in launch_instance

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelapp.py", line 739, in start

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tornado\platform\asyncio.py", line 205, in start

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 608, in run_forever

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 1936, in _run_once

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\events.py", line 84, in _run

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 545, in dispatch_queue

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 534, in process_one

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 437, in dispatch_shell

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\ipkernel.py", line 362, in execute_request

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 778, in execute_request

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\ipkernel.py", line 449, in do_execute

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\zmqshell.py", line 549, in run_cell

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3077, in run_cell

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3132, in _run_cell

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\async_helpers.py", line 128, in _pseudo_sync_runner

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3336, in run_cell_async

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3519, in run_ast_nodes

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3579, in run_code

  File "C:\Users\deanc\AppData\Local\Temp\ipykernel_21724\1451080196.py", line 1, in <module>

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 562, in predict

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 259, in one_step_on_data_distributed

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 249, in one_step_on_data

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 104, in predict_step

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\layer.py", line 908, in __call__

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\operation.py", line 46, in __call__

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 156, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\sequential.py", line 213, in call

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\functional.py", line 182, in call

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\function.py", line 171, in _run_through_graph

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\functional.py", line 637, in call

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\layer.py", line 908, in __call__

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\operation.py", line 46, in __call__

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 156, in error_handler

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py", line 146, in call

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\numpy.py", line 168, in add

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\sparse.py", line 493, in sparse_wrapper

  File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\numpy.py", line 60, in add

Matrix size-incompatible: In[0]: [32,15], In[1]: [210,128]
	 [[{{node sequential_151_1/dense_935_1/BiasAdd}}]] [Op:__inference_one_step_on_data_distributed_9572532]

Don't understand why it's not working anymore, going to skip this code because it's not worth my fucking time no more.

In [ ]:
ywaist_pred = modelwaist_49.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
No description has been provided for this image
In [ ]:
ywaist_pred = modelwaist_50.predict(Xwaist_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
No description has been provided for this image

Despite the increase of 14 data points at the end, the missing last bit is definitely influential to the model's risk! Having the sequential/window period data gives the model a lot of information to learn patterns. We'll revisit the show_score function to work with waist_sequence data.

In [ ]:
show_scores(modelwaist_49, ywaist_sequence_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_sequence_test, expandX_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 39ms/step
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
Out[ ]:
(('Waist Evaluation',
  {'Test MAPE (%)': np.float64(134.8436578920531),
   'Test MSE': 8499.11331141435,
   'Test R^2': -0.2845085176455995}),
 ('Waist Evaluation',
  {'Test MAPE (%)': np.float64(163.04135949807323),
   'Test MSE': 10209.751725000777,
   'Test R^2': -0.5430448534198851}))
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)


ywaist_pred = modelwaist_50.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

The MAPE of the model performs worse than models that don't predict, but want the variations and risk of it. So we'll keep MODEL_49

MODEL_51 - Patience 75¶

Gonna increase patience, but keep delta the same.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_51 = keras.Sequential([
    layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=75,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_51.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=150),
                metrics=['mae'])

hist_51 = modelwaist_51.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [ ]:
plt.plot(hist_51.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [ ]:
show_scores(modelwaist_51, ywaist_sequence_test, expandX_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
Out[ ]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(126.9631042586312),
  'Test MSE': 8542.257855992337,
  'Test R^2': -0.29102914314731354})

MODEL_52 - Extra neuron layer¶

Tryiing another extra layer with more neurons, to see if it further helps with accuracy.

In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_52 = keras.Sequential([
    layers.Dense(256, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(128), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=75,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_52.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=150),
                metrics=['mae'])

hist_52 = modelwaist_52.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_52.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
show_scores(modelwaist_52, ywaist_sequence_test, expandX_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
Out[ ]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(118.76945109124502),
  'Test MSE': 7990.06780642193,
  'Test R^2': -0.20757422308174012})
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber

# set seed
tf.random.set_seed(35)

expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# create model including relu and linear
modelwaist_52 = keras.Sequential([
    layers.Dense(256, input_shape=(expandX_train.shape[1],)), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(128), 
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(64),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    layers.Dropout(0.2),

    layers.Dense(32),
    layers.BatchNormalization(),
    layers.Activation('tanh'),
    
    layers.Dense(1, activation='linear')
])

reduce_lr = ReduceLROnPlateau(monitor='loss',  # Track loss (can also use 'val_loss')
                              factor=0.5,      # Reduce LR by half when triggered
                              patience=75,     # Wait 10 epochs before reducing
                              min_lr=1e-6)     # Do not reduce below this

modelwaist_52.compile(optimizer=Adam(learning_rate=0.001),
                loss=Huber(delta=150),
                metrics=['mae'])

hist_52 = modelwaist_52.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

To be honest, from this point onwards, there's just very little incremental changes, that copy pasting and changing one singular line of code is taking up way too much space on this work book. So I'll be compiling all the changes I've made in order, why I made those changes, and will note any significant details.

  • MODEL_55

    # ADDED code
    layers.Dense(256, input_shape=(expandX_train.shape[1],),
             kernel_regularizer=regularizers.l2(0.01)),  # Add L2 regularization

    Chatgpt suggested L2 regularization makes the model more flexible, and does show in the loss history

  • MODEL_56

    # ADDED code
layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.1),

    The 256 neuron layer did slightly better with the added complexity, so made another layer on top of that to 512. It created spikes of loss increases, but comes back down, though still improves loss overall

  • MODEL_57

    # TRIED code
optimizer=Adam(learning_rate=0.01)

    Increase learning rate of Adam to allow model to add more variance. Model stagnated around loss 500 between epoch 200-500 and didn't have extent of improvement as 56

  • MODEL_58

    # ADDED code
# change all dropout to 0.1
layers.Dropout(0.1),

    All remaining Dropout set from 0.2, to 0.1. Makes model more prone to overfitting, and increases it's risk taking. Overfitting is taking effect slightly with decrease in loss

  • MODEL_59

    # TRIED code
def cyclic_lr(epoch, lr):
base_lr = 1e-4
max_lr = 1e-2
step_size = 2000
cycle = np.floor(1 + epoch / (2 * step_size))
x = np.abs(epoch / step_size - 2 * cycle + 1)
lr = base_lr + (max_lr - base_lr) * np.maximum(0, (1 - x))
return lr

# Create the callback
lr_scheduler = LearningRateScheduler(cyclic_lr, verbose=1)

    Try Cyclic Learning Rate (CLR). Rather than having lr going down, this one goes up and down in a specific range of values for lr. Something like sine waves. Heavy increase spike of loss and lack of improvement

  • MODEL_60

    # ADDED code
modelwaist_53.fit(expandX_train, ywaist_sequence_train, batch_size=64, epochs=1000, verbose=0, callbacks=[reduce_lr])

    Batch size is used to define how many training examples are used to compute the gradient/predictions. There's some slight improvement

  • MODEL_61

    # TRIED code
patience=20

    ChatGPT suggested that patience 75 is too high, and try something between 15 to 30. So Tried 20. Sopped the random spike of loss increase, but loss is not as low as before

  • MODEL_61

    # TRIED code
scaler = StandardScaler()
expandX_train = scaler.fit_transform(expandX_train)

    Feature engineer by normalizing training/testing data. Had similar loss, but had trouble with predicting data on test data, with MAPE reaching +200

  • MODEL_62

    # ADDED code
layers.LeakyReLU(alpha=0.1),

    Trying a different activation function with LeakyReLU. Had success with it along with Tanh. Was really good at decreasing Loss, but seem to overfit too much as MAPE was at 200 for test data

  • MODEL_63

    # ADDED code
layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.001)),
layers.Dropout(0.3),

    Due to overfitting, I wanna decrease it a bit. ChatGPT suggests reducing l2 regularizer, and increase Dropout to improve regularization. Still same issue, where improves really fast, and then stops improvement quite drastically

  • MODEL_64

    # TRIED code
early_stopping = EarlyStopping(monitor='val_loss', patience=100, restore_best_weights=True)

    Adding early stopping to prevent the model from overfitting, after it stops improving. Didn't really activate so didn't do its job

  • MODEL_65

    # ADDED code
layers.ActivityRegularization(l2=0.01),

    Another activation function, which prevents overfitting, discouraging overly large activations. Similar output, and the graph visual looks eerily similar to the true data, except being offsetted by 1 or 2 days

  • MODEL_66

    # REMOVED code
batch_size=64

    Thought this code might cause some sort of data leakage, and wanna see results if removed. No change

Data leakage?¶

I'm wondering there's some sort of data leakage going on, as it looks like the predicted data is offset from the true data by a few points. So I'll be running a few code to have a look. Firstly, I'll check if there's lag on prediction.

Code has returned that there is a lag of -2, where it's predicted about 2 indexes on average after the true data. My next step, is looking at data importance. So there are these two specific features that cause very positive and very negative correlation to each other, but the data it's trained on expandX_train has 210 different features, likely because of the sequence data of a moving 14 day average or some shit. So I have no idea how to really interpret the graph, other than some feature causing the model some heavy reliance upon it.

I think I'll once again remove sequence data and see how the model plays out.

In [ ]:
# Check prediction lag

import numpy as np
import matplotlib.pyplot as plt

# Flatten the data before cross-correlation
true_data = ywaist_sequence_test.flatten()
predicted_data = ywaist_pred.flatten()
# Compute cross-correlation
def cross_correlation(true_data, predicted_data):
    correlation = np.correlate(true_data, predicted_data, mode='full')
    lag = np.argmax(correlation) - len(true_data) + 1
    return lag

# Compute cross-correlation
lag = cross_correlation(true_data, predicted_data)
print(f"Optimal lag is: {lag}")

Optimal lag is: -2
In [ ]:
import lime
from lime import lime_tabular

# Initialize LIME Explainer
explainer = lime_tabular.LimeTabularExplainer(expandX_train, training_labels=ywaist_sequence_train, mode='regression')

# Choose a data point to explain
i = 42  # index of the data point to explain
exp = explainer.explain_instance(expandX_test[i], modelwaist_53.predict)

# Visualize the explanation
exp.show_in_notebook()

157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)


ywaist_pred = modelwaist_53.predict(expandX_test)

fig, ax = plt.subplots(figsize=(20,10))

ax.set(title='Waist measurement difference prediction', xlabel='Date')

ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
No description has been provided for this image

This is the kind of result's I've been seeing so far with experimenting

  • MODEL_67

    # REMOVED code
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
    • See if this sequence data of expanding X_train causes the data leakage. And yup it seems to be the case, as loss overtime was decrease at a more constant pace, rather than a sharp drop off close to 0

  • MODEL_68

    # ADDED code
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)

# CHANGED code to
def create_sequences(X, y, window_size):
    Xs, ys = [], []
    for i in range(len(X) - window_size):
        Xs.append(X[i:i+window_size])
        ys.append(y[i])  # Fixed code
    return np.array(Xs), np.array(ys)
    • Had ChatGPT, help form me a new sequence function that will result in no lag and no data leakage
    ys.append(y[i:i+window_size]) # OLD
ys.append(y[i]) # NEW
    • With the y sequence i+window_size, this basically stated the y value on the index, plus the window_size. Meaning we get a re-formatted y data, wherethe model is predicting the future value, instead of the current value. This makes it easy for the model to predict, as the data is leaked from X, because it has access to future data that it wasn't supposed to have
    • After the change, the model now makes reasonable predictions!

  • MODEL_69

    # CHANGED code
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.3),
layers.ActivityRegularization(l2=0.01),

layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.ActivityRegularization(l2=0.01),
    • Removed Batch Normalization layers from the last two Dense layer. ChatGPT says it might cause distortions on final predictions. Doesn't seem to affect much on the loss or prediction

  • MODEL_70

    # CHANGED code
layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.3),

layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.3),

layers.Dense(128), 
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.3),
    • Removed ActivityRegularization layers where there are batch normalzation layers. It's likely not nessecary as dropout already does regularization. Prediction does seem to slightly be more riskier, and shows loss improvement

  • MODEL_71

    # CHANGED code
layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)),
layers.Dropout(0.1),
    • Reduce dropout, and increase regularizer l2. These were changed because of overfitting data, but because of sequence function data leak and fixing it up, it was underfitting and not making risky predictions. It's now overfitting heavily

  • MODEL_72

    layers.Dropout(0.2)
    • Increase dropout to reduce overfitting, and shows slight improvement

  • MODEL_73

    # CHANGED code
loss=Huber(delta=50)
    • Reduced Huber, model prediction is all over the place. It's being risky, but doeszn't seem to predict the data well

  • MODEL_74

    # ADDED code
hist_53 = modelwaist_53.fit(expandX_train, ywaist_sequence_train, 
                        validation_split=0.2, 
                        epochs=1000, verbose=0, 
                        callbacks=[reduce_lr])
    • Added validation split, set to 0.2. 80% Training, and 20% Validation on the training data provided. Seem to improve model training, but flattens very quickly. Looks like it's overfitting

  • MODEL_75

    # CHANGED code
epochs=500
    • The model has been getting really good at overfitting, and epoch flattens out considerably. Prediction is slightly better

  • MODEL_76

    # CHANGED code
patience=50
    • Wanted to see what changes, if reduced from 75 to 50. During test data graph, it seemed like the prediction values are pulled down by a considerable margin from the true values. Say if prediction is 100, it would be -200 which is strange

  • MODEL_77

    # ADDED code
scaler = StandardScaler()
expandX_train = scaler.fit_transform(expandX_train)
ywaist_sequence_train = scaler.fit_transform(ywaist_sequence_train.reshape(-1, 1)) # reshape from 1D to 2D, as scaler only accepts 2D
    • ChatGPT suggested to try scaling values as it usually helps with prediction. The model did pretty well with predictions around the middle, but struggle with the outliers. MAPE which is what I was using to judge prediction accuracy for each model, is now messed up as its 588 instead of the rough 130-200 the model was getting. Likely due to the scaler changing y axis values

  • MODEL_78

    # CHANGED code
layers.Activation('tanh'),
    • Still remembered tanh do well with values around 0, and the scaler should be perfect for tanh. It learned training data with zero effort, immediately flattening out the epoch, but did poorly at predictions, likely due to overfitting

  • MODEL_79

    # CHANGED code
epochs=100
    • Due to how quick the model is learning patterns from the training data, I've further reduced epochs to 100 to prevent overfitting. Even though loss is close to 0 and has loss all meaning of whether the loss over epoch graph is indicative to the model's performance, the prediction is very poor, showing very little risk taking from the model

  • MODEL_80

    # CHANGED code
layers.LeakyReLU(alpha=0.1)
epochs=1000
    • I returned to 1000 epochs, realizing despite the epoch loss flattens, it doesn't mean a consistent result for the model prediction. Teetering back to LeakyReLU as well. With 1000 epochs, it does show the model taking the proper risks

  • MODEL_81

    # CHANGED code
layers.Activation('tanh'),
    • Again coming back to tanh, just to see it's prediction with 1000 epochs. MAPE has improved, but is not risking prediction values. We'll stick with LeakyReLU from now on

  • MODEL_81

    # ADDED/CHANGED code
layers.Dense(1024, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),

layers.Dense(512),
    • Create an even denser model to see if it can improve performance. Doesn't seem to make much improvement

  • MODEL_82

    # ADDED code
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate=0.001,
decay_steps=1000,
decay_rate=0.9
)

modelwaist_53.compile(optimizer = Adam(learning_rate=lr_schedule),...)
    • Applied exponential learning rate decay. It's used to help the model learn quickly at first with a higher lr, and has better stability as lowering lr precents overshooting of optimal solution

  • MODEL_83

    # USED code
import numpy as np
errors = np.abs(expandX_test - ywaist_sequence_test)  # Compute absolute errors
print(np.percentile(errors, [25, 50, 75, 90]))  # Look at error percentiles

# RESULT
[0.33977055 0.68632555 1.34403686 1.84173002]

loss=Huber(delta=1.8),
    • What the code above does, is telling me the percentile of where the error occurs. 90, means 90% of my errors are at 1.8 unit or below. 50, is 50% of error at 0.6 unit or below. This helps indicate the delta I should use for huber. As my data has changed from x100 of original dataset, to now scaled, the delta should change accordingly to match the value distribution of my new data. So I would set my Delta to 1.8

  • MODEL_84

    # CHANGED code
layers.Dense(256, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)),

# REMOVED code
layers.Dense(512),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),

layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
    • I've deleted two layer bunches, as the model seems to overfit a lot on the training data, and not exactly make good predictions

  • MODEL_85

    # CHANGED code
decay_steps=500,

# REMOVED code
reduce_lr = ReduceLROnPlateau(monitor='loss',
                          factor=0.5,    
                          patience=100,   
                          min_lr=1e-6)
    • Exponential decay might be too slow, so we'll have it reduce the learning rate at a faster speed. Also encountered an error where both exponential decay and reduce_lr conflict each other, with how they change lr. So removed reduce_lr, to see how exponential decay works

  • MODEL_86

    # ADDED code
layers.GaussianNoise(0.05, input_shape=(expandX_train.shape[1],)),
    • Adding a layer of gaussian noise, which can help prevent model from over relying on exact training values. Should help it generalize better

  • MODEL_87

    # CHANGED code
loss=tf.keras.losses.LogCosh()
    • Replaced Huber with LogCosh. This loss function allows the model to take bigger prediction steps without being heavily punished, e.g. outlier points

  • MODEL_88

    # REMOVED code
layers.ActivityRegularization(l2=0.01),

# CHANGED code
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),

layers.Dense(256), 
layers.Dense(128),
layers.Dense(64),
    • Activity Regularization is removed, as it performs a similar function to batch normalization, and having a lot of these kinds of layers especially near the end, gets redundant. I've also moved Density of neurons up by one exponent, and shrunk regularizer.l2 from 0.01 to 0.001, as it could be too strong and only predicting conservatively

  • MODEL_89

    # CHANGED code
layers.Dropout(0.6),

decay_steps=200,

    • Seems like I'm reaching a wall in terms of prediction, as it seems to not really have much affect on prediction, likely due to how there's many more factors that also play on weight change from a day by day basis, such as water weight, sodium intake (which affects amount of water weight), timing of food intake, exercise, etc.

    • I'm just curious what a very high dropout, and very low decay step for lr_schedule is going to do

  • MODEL_90

    # CHANGED code
layers.GaussianNoise(0.1, input_shape=(expandX_train.shape[1],)),

layers.Dropout(0.1),
    • Put dropout down to 0.1, as it stops learning to a point, and seems to make very cautious predictions. Gaussion is increased from 0.05 to 0.1

  • MODEL_91

    # ADDED code
from tensorflow.keras import backend as K

def weighted_log_cosh(y_true, y_pred):
error = y_pred - y_true # calculate difference/error of pred and truth
weight = K.exp(K.abs(error)) # apply exponents to absolute error, creating more emphasis on large errors
return K.mean(weight * K.log(tf.cosh(error + 1e-12))) # tf.cosh(error) > cosine of the error, used for smoothing approximation of error, K.log() > smoothe the penalty,1e-12 > very small value added to avoid calculations at 0
    • This block of code, helps penalize under predictions, which should get it to predict more riskier, and it somewhat did what was advertised

  • MODEL_92

    # CHANGED code

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6  # Minimum LR
)
    • lr_scheduler is changed to CosineDecayRestarts. Code forces model to restart learning at periodic intervals, escaping local minima, and encourages bolder updates throughout training

  • MODEL_93

    # CHANGED code
layers.Dense(1, activation='tanh')
    • I've used tanh on LeakyReLU layers, but never on the last linear layer. So let's give that a go. Not the best results, as prediction laid between 1 and -1, but standard deviation values have reach numbers past 3 and -3

  • MODEL_94

    # CHANGED code
layers.Dense(1)
    • Changed it to not include tanh or linear. Model has been encouraged to make riskier predictions slightly

  • MODEL_95

    # CHANGED code
weight = K.log(1 + K.exp(K.abs(error)))
    • ChatGPT suggested change to weight_log_cosh function, as it was penalizing extreme predictions too harshly, and this should help, by decreasing/softening the weight of extreme values. It is taking more risks now, and the loss error graph stopped showing how it's reaching 0 loss immediately, but gradually instead

  • MODEL_96

    # REMOVED code
layers.Dropout(0.1)
    • Curious as to see the results, predictions became a bit reserved

  • MODEL_97

    # CHANGED code
weight = K.log(5 + K.exp(K.abs(error)))
    • k.log(was 1), now it is 5. This is done as previously stated, we wanted to soften the punishment on extreme values. And having that 1, basically elevated all smaller value with a +5, making the errors of the extreme value even smaller. Model is showing to take risks more

  • MODEL_98

    # ADDED code
def quantile_loss(q):
def loss(y_true, y_pred):
    e = y_true - y_pred # calculate error difference between pred and truth
    return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss

loss=quantile_loss(0.9),

# REMOVED code
def weighted_log_cosh(y_true, y_pred):
    error = y_pred - y_true # calculate difference/error of pred and truth
    weight = K.log(5 + K.exp(K.abs(error))) # apply exponents to absolute error, creating more emphasis on large errors
    return K.mean(weight * K.log(tf.cosh(error + 1e-12))) # tf.cosh(error) > cosine of the error, used for smoothing approximation of error, K.log() > smoothe the penalty,1e-12 > very small value added to avoid calculations at
    • gpt suggested to try another loss function called quantile_loss. It has a value called q, where you can change to penalize either underpredicton (higher value), or overprediction (lower value) more. So i chose 0.9 as the starting point. Some of the riskiest prediction I've seen!0

  • MODEL_99

    # CHANGED code
loss=quantile_loss(0.7),
    • First time saying that the model is playing a bit risky, and need to tone down a bit.

Final Verdict on Model¶

After 99 iterations of the model, I think I've found what is the most ideal model. It's not the best prediction results, likely due to how daily changes to waist and weight are affected by more than just current weight/waist, macros and calories.

Here's is the fixed sequence function, where the y data is changed to not have any data leak (something I've mentioned a while ago).

In [188]:
# Adjust by removing the lag by setting ys.append(y[i]) instead of y[i+window_size]
def create_sequences(X, y, window_size):
    Xs, ys = [], []
    for i in range(len(X) - window_size):
        Xs.append(X[i:i+window_size])
        ys.append(y.iloc[i])  # Fix the lag by predicting current time step. It predicts the first value of the window_size, rather than a future value, since we added the window_size to the index.
    return np.array(Xs), np.array(ys)

window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)

Final Model¶

In [190]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
from tensorflow.keras import regularizers
from tensorflow.keras.optimizers import Adam



# Set seed for reproducibility
tf.random.set_seed(35)

# Ensure input shape is correct
Xwaist_sequence_train = Xwaist_sequence_train.reshape(Xwaist_sequence_train.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train = scaler_X_waist.fit_transform(Xwaist_sequence_train)
scaler_y_waist = StandardScaler()
normY_waist_train = scaler_y_waist.fit_transform(ywaist_sequence_train.reshape(-1, 1))

# Define model
modelwaist = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1) 
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.001,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelwaist.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.7),
                metrics=['mae'])

hist = modelwaist.fit(normX_waist_train, normY_waist_train, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(

The loss history of training the model, after 1000 epochs¶

In [191]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[191]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image

Showing scores of model¶

In [192]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))


show_scores(modelwaist, normY_waist_test, normX_waist_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
Out[192]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(403.76744447706216),
  'Test MSE': 1.0789613768209771,
  'Test R^2': -0.14378946955523975})

Monte Carlo predictions¶

What this function does, is it simulates a number of predictions on the test results. It returns the mean prediction, along with the range of 2 standard deviation.

In [193]:
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_waist, uncertainty = monte_carlo_predictions(modelwaist, normX_waist_test, n_simulation=100)

Model Prediction vs Truth Graph¶

I'll create a graph, to display the true values of waist difference, along withmonte carly predictions (mean prediction + 2 standard deviation)

In [194]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_waist)), mean_pred_waist, label="Waist Prediction", color='blue')
plt.plot(normY_waist_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist)), mean_pred_waist - 2 * uncertainty, mean_pred_waist + 2 * uncertainty, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image

Weight Prediction¶

Now that we've got the prediction of Waist, I tried it on the Weight data.

I did some fixing up with variable names and doing some cleaning/reorganizing of code. However it's underpredicting compared to the previous model with the unorganized variable names. So will list the changes like the waist model to better model:

  • MODEL_1

    # CHANGED code
epochs=100
    • Change epoch=1000 to 100. This is so testing and visualizing prediction trends be very quick. Unadvertedly made the model take more risks, so will be playing with epochs

  • MODEL_2

    # CHANGED code
epochs=200
    • 100 to 200 epochs to see if risk is increasing

  • MODE_3

    # CHANGED code
loss=quantile_loss(0.9),

epochs=300
    • 200 to 300 epochs, and increase quantile_loss to hopefully promote more risk taking, not much change

  • MODEL_4

    # CHANGED code
loss=quantile_loss(0.1),
    • Minimize quantile_loss to see results. Underpredicted

  • MODEL_5

    # CHANGED code
initial_learning_rate=0.005,

loss=quantile_loss(1),
    • *Higher learning rate to help with larger updates on weights over tiny ones, and set quatile loss at highest, to 1. Prediction is somehow made around value 10 over 0

  • MODEL_6

    # CHANGED code
loss=quantile_loss(0.9),
    • Reduce quantile_loss back to 0.9, model predicts as it should

  • MODEL_7

    # CHANGED code
epochs=1000
    • Increase epoch back to 1000, seems like model's trajectory continues treading down. Model takes more risks but seems a bit erratic in prediction

  • MODEL_8

    # CHANGED code
layers.Dropout(0.3),
    • Increase dropout to see if it helps with erratic prediction and generalization, model is not predicting negative values

  • MODEL_9

    # CHANGED code
initial_learning_rate=0.001,

layers.Dropout(0.2),
    • *Reduced dropout, and reused the same initial learning_rate since it may have cause such prediction troubles. Now I realize if initial lr is high, it artificially pulls up the prediction

  • MODEL_10

    # CHANGED code
initial_learning_rate=0.0005,
    • *Reduced initial lr to prevent artificial pulling up of prediction values

  • MODEL_11

    # ADDED code
layers.Dense(1024, kernel_regularizer=regularizers.l2(0.001)),  # More neurons
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
    • Creating another dense layer batch, with 1024 neurons. Moving regularizers l2 to the 1024 layer, performance was mediocre

  • MODEL_12

    # CHANGED code
layers.Dense(512, kernel_regularizer=None),
    • kernel_regularizer set to none, chatgpt suggests the l2 regularizer may constrain model's ability to aggressively explore patterns, aka take more risks. Changes not significant

  • MODEL_13

    # CHANGED code
layers.Dropout(0.1),
    • setting dropout back to 0.1, again lack of change
In [122]:
Xweight_sequence_train, yweight_sequence_train = create_sequences(Xweight_train, yweight_train, window_size)
Xweight_sequence_test, yweight_sequence_test = create_sequences(Xweight_test, yweight_test, window_size)
In [131]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K


# Set seed for reproducibility
tf.random.set_seed(35)

# Ensure input shape is correct
Xweight_sequence_train = Xweight_sequence_train.reshape(Xweight_sequence_train.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train = scaler_X_weight.fit_transform(Xweight_sequence_train)
scaler_y_weight = StandardScaler()
normY_weight_train = scaler_y_weight.fit_transform(yweight_sequence_train.reshape(-1, 1))

# Define model
modelweight = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=None),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1)
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.0005,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelweight.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.9),
                metrics=['mae'])

hist = modelweight.fit(normX_weight_train, normY_weight_train, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [132]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[132]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [165]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))


show_scores(modelweight, normY_weight_test, normX_weight_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step
Out[165]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(370.2774930790802),
  'Test MSE': 2.08010199127118,
  'Test R^2': -0.13372795895491119})
In [166]:
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_weight, uncertainty_weight = monte_carlo_predictions(modelweight, normX_weight_test, n_simulation=100)
In [167]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_weight)), mean_pred_weight, label="Weight Prediction", color='blue')
plt.plot(normY_weight_test, label='True weight', color='red')
plt.fill_between(range(len(mean_pred_weight)), mean_pred_weight - 2 * uncertainty_weight, mean_pred_weight + 2 * uncertainty_weight, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")

plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image

Results are okay, I mean it seems to underfit a bit especially near the latter portion of the data, but I say it works fine for now.

Model performance on training data???¶

I'm bummed out how hard it is to get the model to take risks in its predictions, and don't understand how many tries in perfecting hyperparameters, layers, and functions, it doesn't make too much of a dent in getting the model to risk take more.

So now I'm curious, what does the model look like if I use the model to predict the training data instead, and compare it to the actual train labels.

In [136]:
# mean model prediction of waist train
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_waist_train, uncertainty = monte_carlo_predictions(modelwaist, normX_waist_train, n_simulation=10)
In [137]:
# graph plot of waist training data
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(normY_waist_train, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_train)), mean_pred_waist_train - 2 * uncertainty, mean_pred_waist_train + 2 * uncertainty, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_waist_train)), mean_pred_waist_train, label="Waist Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image

Oh look! There's definitely some anomaly as the model continues down the data, especially from 1200 index and beyond, where the model takes a much more cautious prediction.

Now lets see the weight trianing data

In [168]:
# mean prediction weight train
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_weight_train, uncertainty_weight = monte_carlo_predictions(modelweight, normX_weight_train, n_simulation=10)
In [169]:
# weight train graph display
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(normY_weight_train, label='True weight', color='red')
plt.fill_between(range(len(mean_pred_weight_train)), mean_pred_weight_train - 2 * uncertainty_weight, mean_pred_weight_train + 2 * uncertainty, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_weight_train)), mean_pred_weight_train, label="Weight Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image

Jesus, it's even more prominent in the weight data! The underprediction range of the data, is similar to the one in the waist data, starting around index 1200.

Let's see the actual waist values, aka the real waist size day by day rather than the waist diffference, when compared to the prediction values of the model. I'll visualize the waist size, from index 800 to 1500. Even though the anomaly starts around index 1200, I also want to see a number of previous indexes before the anomaly to see if the waist data provides any visual cues towards the anomaly.

In [140]:
# Plotting the waist values between indexes 800-1500
import matplotlib.pyplot as plt

plt.plot(clean_df['Waist (cm)'][800:1500])
plt.title('Raw Waist Values - Index 800-1500')
plt.xlabel('Index')
plt.ylabel('Waist')
plt.legend()
plt.grid(True)
plt.show();

C:\Users\deanc\AppData\Local\Temp\ipykernel_53232\3308172792.py:8: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
  plt.legend()
No description has been provided for this image

We can see where the model underpredicts, is around the period of time where the waist data transitions from stablility, to a persistent increase. Let's see how the other data from our X data compare between index 800 to 1500 as well.

Namely we'll look at weight and calorie consumption, then display weight prediction graph as a comparison as well. (Not macros, as from previous visualizations, they don't have as heavy affect on waist/weight change compared to caloric intake)

In [170]:
# display weight data, caloric consumption, and pred/true values of weight change
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)

ax[0].set(title='Weight value', xlabel='Index')
ax[0].plot(clean_df['Weight (kg)'][800:1500], label='Weight data')

ax[1].set(title='Caloric consumption', xlabel='Index')
ax[1].plot(clean_df['Daily intake (kcal)'][800:1500], label='Daily kcal intake')

ax[2].set(title='Model prediction vs weight fluctuation', xlabel='Index')
ax[2].plot(range(800,1450), normY_weight_train[800:1450], label='True values') # (range) to only select data of variable from 800-1500. If i did what i did with ax[1]/[0], it'll cut the front out, and bring everything to index 0
ax[2].plot(range(800,1450), mean_pred_weight_train[800:1450], label='Model prediction')
ax[2].legend();
No description has been provided for this image

Yup, it does seem like the change in weight/waist increase and the increase in caloric consumption around index 1200, seem to cause the model to underpredict from there on, and I can see why.

Change in lifestyle:¶

For the most part from index 0 to 1200, my lifestyle was usually very sedentary, and was very home bound. What changed from 1200 was I started to go to the gym 5-6 times week, being a lot more active in addition to eating a lot of food. Lot of these were very different from my old lifestyle, which is probably what my model picked up on during learning.

A lot of my data from the start made sense to it, due to the predictability of how much food I ate, the macronutrients, weight/waist variations etc. But then suddenly it hits a wall at around index 1200 when my caloric consumption was unusual from before, macros were unusual from before and for the most part, my weight and waist weren't following the patterns it was recognizing from the start.

What now?¶

Well, unfortunately there isn't much I can really do with it unless I collect enough data of my weight, waist etc. post lifestyle change, so the model has enough data points to learn the new patterns for it.

And once enough data is collected, I can create two separated data. One with my sedentary lifestyle, and the other with my active lifestyle, so the respective models for each data set can learn the patterns that relate to my current lifestyle in the data.

BUT! Because I spent so much time in perfecting the model, we might as well give it a go, and only train the ml model with the active lifestyle portion of the data to see if the resulting prediction is better.

Retrain model on active lifestyle data portion:¶

In [201]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K


# Set seed for reproducibility
tf.random.set_seed(35)

# Filter training data to start post index 1150, aka where active lifestyle starts
Xwaist_sequence_train_active = Xwaist_sequence_train[1150:]
ywaist_sequence_train_active = ywaist_sequence_train[1150:]

# Ensure input shape is correct
Xwaist_sequence_train_active = Xwaist_sequence_train_active.reshape(Xwaist_sequence_train_active.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train_active = scaler_X_waist.fit_transform(Xwaist_sequence_train_active)
scaler_y_waist = StandardScaler()
normY_waist_train_active = scaler_y_waist.fit_transform(ywaist_sequence_train_active.reshape(-1, 1))

# Define model
modelwaist_active = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1)
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.001,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelwaist_active.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.7),
                metrics=['mae'])

histwaist_active = modelwaist_active.fit(normX_waist_train_active, normY_waist_train_active, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [202]:
plt.plot(histwaist_active.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[202]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [203]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))


show_scores(modelwaist_active, normY_waist_test, normX_waist_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 26ms/step
Out[203]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(235.0922096252269),
  'Test MSE': 1.6186943022889198,
  'Test R^2': -0.19643115107327835})
In [204]:
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_waist_active, uncertainty_active = monte_carlo_predictions(modelwaist_active, normX_waist_test, n_simulation=100)
In [205]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_waist_active)), mean_pred_waist_active, label="Waist Prediction", color='blue')
plt.plot(normY_waist_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_active)), mean_pred_waist_active - 2 * uncertainty_active, mean_pred_waist_active + 2 * uncertainty_active, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image

And this is the prediction on the entire dataset:

alt text

Doesn't seem to vary too much except the active model seem to underpredict a little more than the whole model. Despite that, it's still quite impressive how the active model can make such predictions with less than a year of data, when compared to whole model, with almost 5 times as much data as active.

Active model prediction on training data:¶

Since we've had a look at model's prediction on the whole training data, let's see the active model's prediction on it's training data.

In [335]:
# mean prediction of waist on the active data
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_waist_active_train, uncertainty_waist_train = monte_carlo_predictions(modelwaist_active, normX_waist_train_active, n_simulation=10)
In [336]:
mean_pred_waist_active_train.size
Out[336]:
306
In [339]:
# active train graph display
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(normY_waist_train_active, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_active_train)), mean_pred_waist_active_train - 2 * uncertainty_waist_train, mean_pred_waist_active_train + 2 * uncertainty_waist_train, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_waist_active_train)), mean_pred_waist_active_train, label="Waist Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()

plt.figure(figsize=(20,5))
plt.plot(clean_df['Waist (cm)'][1150:1150+306], label='True waist', color='red') # why +306 at last index? because that's how long the prediction data is, and would need to match it to get better visual between graph comparison
plt.xlabel("Sample Index")
plt.ylabel("Kilogram")
plt.title("Weight measurement overtime")
plt.legend()
plt.show()

plt.figure(figsize=(20,5))
plt.plot(clean_df['Daily intake (kcal)'][1150:1150+306], label='Calorie', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Kilocalories")
plt.title("Consumption of kcal overtime")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

Hmmmm... still the underprediction exists in the graph despite the data is inclusive of my active lifestyle :(

The last hypothesis for this could be because the model needs more training. Remember from the RNN data? There was still room for improvement in the prediction data, where there's notable spots that the model hasn't predicted, and first did parts that I assume are most notable to the model, then working it's way down to the more nuanced data points.

alt text

Regardless, I think it's a good idea to start wrapping it up, and see how the model's prediction compares once we convert it to day by day weight/waist measurements, to the actual measurements rather than the difference.

Unstandadized data¶

Since I've standardized the value of the train and test value, I'll need to reverse the process. This is so instead of having values of the weight and waist difference, I have a graph comparison on my weight, interchanged to my prediction, and comparing it to my true weight and waist, to see how it does overall with my actual weight and waist prediction.

The data was also multiplied by 100 times before, to make the model learn easier, rather than values sticking around 0.

Standardization errors¶

So I checked the results after reversing standardization, and realized an issue where the data that's reverse, did not match the actual true values at all. A little quick consultation on chatGPT said that, I should be creating separate scales for each y, x, train, test, weight, and waist data.

Normalization errors¶

I've gone back and decided to clean up the variable names to the normalization and scalers, as it used to look like this for my waist:

expandX_train = expandX_waist_train.reshape(expandX_train.shape[0], -1)
scaler_X = StandardScaler()
expandX_train = scaler.fit_transform(expandX_train)
scaler_y = StandardScaler()
y_sequence_train = scaler_y.fit_transform(y_sequence_train.reshape(-1, 1))

No labelling that it is waist data, and reusing same variables for scaling. Not the most professional/cleanest as there are reuse of same variables that might cause problems with trying to reverse standardization

Changed code:¶

expandX_train_waist = expandX_train_waist.reshape(expandX_train_waist.shape[0], -1)
scaler_X_waist_train = StandardScaler()
normX_waist_train = scaler_X_waist_train.fit_transform(expandX_train_waist)
scaler_y_waist_train = StandardScaler()
normY_waist_train = scaler_y_waist_train.fit_transform(ywaist_sequence_train.reshape(-1, 1))

Fit scalars on Test (Big nono)¶

I created separate scalars of standardization for every data, but I felt something was a bit off with doing what I was trying to do. Chat GPT has said that the scaling of test data, can cause data leakage as I was using fit_transform(). Rather, transform() should be used on the test data, and one scalar be used between the Train and Test sets of either X or y.

Redoing weight model¶

After fiddling with scalars and getting them right, the model is now underpredicting and being too cautious, so will need to do a bit of testing again.

I have redone the code of the final weight/waist model from the above. Now I'm ready to pursue with the rest of the code!

In [158]:
# reverse standardization of the data
y_pred_waist = scaler_y_waist.inverse_transform(mean_pred_waist.reshape(-1, 1)) # change the data into 2D array fr inverse transformation
y_true_waist = scaler_y_waist.inverse_transform(normY_waist_test) # data is already in 2D array

y_pred_weight = scaler_y_weight.inverse_transform(mean_pred_weight.reshape(-1, 1))
y_true_weight = scaler_y_weight.inverse_transform(normY_weight_test)

y values were multiplied by 100 before, so will need to divide back to original values

In [ ]:
y_pred_weight = y_pred_weight/100
y_true_weight = y_true_weight/100

y_pred_waist = y_pred_waist/100
y_true_waist = y_true_waist/100
In [195]:
# see if waist data matches
y_true_waist[:5],ywaist_sequence_test[:5]
Out[195]:
(array([[ 0.8],
        [ 0.6],
        [-0.2],
        [-0.4],
        [ 0.1]]),
 array([ 80.,  60., -20., -40.,  10.]))

Prediction values vs. waist/weight measurements¶

Now I want to see how the model visually does compared to my actual waist/weight values, since it's hard to judge from the fluctuations of weight/waist differences.

Next step is to add the first weight/waist measurment value on to the first indexed prediction, and let it mathematecally operate on the oncoming prediction values.

In [160]:
start_waist = Xwaist_test['Waist (cm)'][:1]
start_weight = Xweight_test['Weight (kg)'][:1]

start_waist, start_weight
Out[160]:
(1463    73.2
 Name: Waist (cm), dtype: float64,
 1463    72.1
 Name: Weight (kg), dtype: float64)
In [161]:
# convert prediction values to prediction of measurement instead
pred_waist_measure = [start_waist] # create a new list, and have starting value at the first waist measurement

for val in y_pred_waist: # create looping function to mathematically operate prediction values onto the first waist measurement
    next_val = pred_waist_measure[-1] + val # get previous value and add/subtract the current value
    pred_waist_measure.append(next_val)

pred_waist_measure = pred_waist_measure[1:] # remove the first value, to keep with same data shape of test

pred_waist_measure = np.array(pred_waist_measure)
pred_waist_measure = pred_waist_measure.flatten()
In [178]:
# convert prediction values to prediction of measurement instead
pred_weight_measure = [start_weight] # create a new list, and have starting value at the first weight measurement

for val in y_pred_weight: # create looping function to mathematically operate prediction values onto the first weight measurement
    next_val = pred_weight_measure[-1] + val # get previous value and add/subtract the current value
    pred_weight_measure.append(next_val)

pred_weight_measure = pred_weight_measure[1:] # remove the first value, to keep with same data shape of test

pred_weight_measure = np.array(pred_weight_measure)
pred_weight_measure = pred_weight_measure.flatten()

Prediction and true measurement comparison¶

In [196]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure)), pred_waist_measure, label="Waist Prediction", color='blue')
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction converted to measurement vs waist measurment")
plt.legend()
plt.show()

plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_weight_measure)), pred_weight_measure, label="Weight Prediction", color='blue')
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label='Weight Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Weight prediction converted to measurement vs weight measurment")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image

Observations:¶

Its clear that the full data model training does a steady continuous incline. The likely reasoning is due to the sedentary lifestyle contained a much heavier weight than the active lifestyle, due to how long I stayed sedentary during my data recording, but is definitely more evident in weight prediction.

So when it came to predicting my weight in the portion where I was in my active lifestyle, the majority of the pattern it has learnt was only the sedentary lifestyle. Naturally that would mean less caloric and macro consumption to keep a stable weight when sedentary, but is vice versa for an active lifestyle. Unfortunately the model did not understand the change in lifestyle, and treat my caloric and macro consumption as if I was still in my sedentary lifestyle.

But also another reasoning for the increase as well, could be seen in the way the model predicts training data. Despite matching well with the positive values, it just didn't do a good enough job with the negative values, which could likely have caused that imbalance.

alt text

Regarding that, it would be interesting to see the model make prediction on the training data with the true measurments, and see if it performs that over-estimation of weight/waist of the data.

In [197]:
# redo inverse standardization with the training data
y_pred_waist_train = scaler_y_waist.inverse_transform(mean_pred_waist_train.reshape(-1, 1)) # change the data into 2D array fr inverse transformation
y_true_waist_train = scaler_y_waist.inverse_transform(normY_waist_train) # data is already in 2D array

y_pred_weight_train = scaler_y_weight.inverse_transform(mean_pred_weight_train.reshape(-1, 1))
y_true_weight_train = scaler_y_weight.inverse_transform(normY_weight_train)

# divide 100 to get original data
y_pred_waist_train = y_pred_waist_train/100
y_true_waist_train = y_true_waist_train/100

y_pred_weight_train = y_pred_weight_train/100
y_true_weight_train = y_true_weight_train/100

start_waist_train = Xwaist_train['Waist (cm)'][:1]
start_weight_train = Xweight_train['Weight (kg)'][:1]


# convert prediction values to prediction of measurement instead
pred_waist_measure_train = [start_waist_train] # create a new list, and have starting value at the first waist measurement

for val in y_pred_waist_train: # create looping function to mathematically operate prediction values onto the first waist measurement
    next_val = pred_waist_measure_train[-1] + val # get previous value and add/subtract the current value
    pred_waist_measure_train.append(next_val)

pred_waist_measure_train = pred_waist_measure_train[1:] # remove the first value, to keep with same data shape of test

pred_waist_measure_train = np.array(pred_waist_measure_train)
pred_waist_measure_train = pred_waist_measure_train.flatten()

# convert prediction values to prediction of measurement instead
pred_weight_measure_train = [start_weight_train] # create a new list, and have starting value at the first waist measurement

for val in y_pred_weight_train: # create looping function to mathematically operate prediction values onto the first waist measurement
    next_val = pred_weight_measure_train[-1] + val # get previous value and add/subtract the current value
    pred_weight_measure_train.append(next_val)

pred_weight_measure_train = pred_weight_measure_train[1:] # remove the first value, to keep with same data shape of test

pred_weight_measure_train = np.array(pred_weight_measure_train)
pred_weight_measure_train = pred_weight_measure_train.flatten()

I also just realized, I could've just created a function to process the inverse standardization rather than copy pasting variable names. There's definitely a LOT of things I could've just moved into functions, and make this job a lot fuckin easier lmao.

In [198]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure_train)), pred_waist_measure_train, label="Waist Prediction", color='blue')
plt.plot(range(len(Xwaist_train)), Xwaist_train['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction converted to measurement vs waist measurment")
plt.legend()
plt.show()

plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_weight_measure_train)), pred_weight_measure_train, label="Weight Prediction", color='blue')
plt.plot(range(len(Xweight_train)), Xweight_train['Weight (kg)'], label='Weight Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Weight prediction converted to measurement vs weight measurment")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image

Some pretty unfortunate results sadly, as hypothesis is proven to be true.

Now I'd say it'd be worth it to also see the results for the active model as well.

In [206]:
pred_waist_active = scaler_y_waist.inverse_transform(mean_pred_waist_active.reshape(-1, 1))
pred_waist_active = pred_waist_active/100

# convert prediction values to prediction of measurement instead
pred_waist_measure_active = [start_waist] # create a new list, and have starting value at the first waist measurement

for val in pred_waist_active: # create looping function to mathematically operate prediction values onto the first waist measurement
    next_val = pred_waist_measure_active[-1] + val # get previous value and add/subtract the current value
    pred_waist_measure_active.append(next_val)

pred_waist_measure_active = pred_waist_measure_active[1:] # remove the first value, to keep with same data shape of test

pred_waist_measure_active = np.array(pred_waist_measure_active)
pred_waist_measure_active = pred_waist_measure_active.flatten()
In [207]:
import matplotlib.pyplot as plt

plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure_active)), pred_waist_measure_active, label="Waist Prediction Active Model", color='blue')
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction active model converted to measurement vs waist measurment")
plt.legend()
plt.show()
No description has been provided for this image

Observations:¶

Graph gives some slightly more favourable/reasonable prediction when it was trained on data that was mostly during my active lifestyle. But still seems to have that steady increase near the end portion of the data.

My thought about that steady increase could be because of another change in my diet. For the most of the data, I was performing bulking during my active lifestyle, and then at a later portion of time within the test data, I started to cut instead of bulking. That change in caloric consumption/macro could've been something that the model wasn't expecting, due to all the major changes in my diets, lifestyle, and fitness.

Use sequential, autoregressive prediction loop¶

One thing I wanted to try with the model, is instead of predicting the change of weight/waist based on given data, inclusive of the previous day's weight/waist measurement, that I instead make the model predict the change of weight/waist size, then on the next day, that prediction change is them added/subtracted from the previous weight/waist data. The model compounds it's prediction over and over.

This is a more realistic approach to such a model, as you wouldn't reasonably have data of every weight/waist measurement from the day before to continually feed the model in order for it to find the difference for the next day. This will show how the model attempts to predict the weight/waist on its own, based on its previous predictions rather than based on a set data table.

There are several important points to note of for this forecasting model...

Simulated Inferece Loop:¶

We have the trained model and test data, now we need a loop function, to continuously loop in the weight/waist change prediction and apply its value upon their respective column inside X_test.

Inverse standardize, then re-standardize:¶

To properly add in the prediction to the starting weight/waist, we will need to make a prediction, inverse standardize the results, as both X and y datasets use different standardization to fit to it, making both datasets incompatable unless reversed standardization. Once appended into the X test, we'll need to restandardize again for the next prediction, as that's the data the model has learnt from.

Simultaneous prediction of Weight and Waist:¶

A thing to note of, is in the X training data, it includes the weight and waist measurement. If I were to do the autoregressive prediction on weight, the same will have to be done on waist at the same time. This is because if you don't have access to future measurments of weight, then you reasonably won't have waist measurments as well. Therefore forecasting both predictions at the same time would be more realistic, than a column that gives away crucial information.

P.S. Typing this out, I realized that the test data between weight and waist is probably the same, as the only real change was their targets (y data). So I will do a boolean operation to see this in action

In [288]:
# checking if both waist and weight test data are the same or not, with 10^-13 tolerance
np.allclose(scaler_X_waist.inverse_transform(normX_waist_test), scaler_X_weight.inverse_transform(normX_weight_test), atol=1e-13)
Out[288]:
True

Yup, indeed the same model. It returns false if atol is 1e-14, which is likely due to standardization taking effect

In [ ]:
# Create sequential, autoregressive prediction loop
def simulation_loop(model_weight, model_waist, norm_X, id_waist, id_weight, X_scaler, 
scaler_y_weight, scaler_y_waist, start_weight, start_waist, steps_ahead):
    """
    Forecasting weight and waist values through compounding predictions
    
    Parameters:
    model_weight, model_waist: trained models
    norm_X: standardized test set
    id_waist, id_weight: indexed location of where waist and weight is located in X_test dataset
    X_scaler, scaler_y_weight, scaler_y_waist: scalers for each respective datasets, to either fit or inverse transform when needed
    start_weight, start_waist: starting value of forecast
    steps_ahead: number of steps to predict/forecast
    """
    current_weight = float(start_weight) # make it float, as to not get rid of decimal predictions
    current_waist = float(start_waist)
    predictions = [] # keep new list for all predictions

    # unstandardize x_test
    X_test = X_scaler.inverse_transform(norm_X)    

    for i in range(steps_ahead):
        # prepare current index of x_test in loop to a new variable
        x = X_test[i].copy()

        # replace real measurements with predictions
        x[id_weight] = current_weight
        x[id_waist] = current_waist

        # reattach the current index row to the test dataset
        X_test[i] = x

        # scale test for model training
        norm_X = X_scaler.transform(X_test)

        # predict weight and waist
        weight_pred_scaled = model_weight.predict(norm_X, verbose=0)
        waist_pred_scaled = model_waist.predict(norm_X, verbose=0)

        # inverse scaling of y predictions (including the *100 done on data)
        weight_pred = scaler_y_weight.inverse_transform(weight_pred_scaled)[i][0]/100
        waist_pred = scaler_y_waist.inverse_transform(waist_pred_scaled)[i][0]/100

        # update values for next loop
        current_weight += weight_pred
        current_waist += waist_pred

        predictions.append((current_weight, current_waist))

    return predictions
In [290]:
# call in function with all details needed to predict weight and waist values with this forecasting function
preds = simulation_loop(modelweight, modelwaist, normX_weight_test, 
                        id_weight=0, id_waist=1, X_scaler=scaler_X_weight, 
                        scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist, 
                        start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
                        steps_ahead=len(normX_weight_test))

forecast_weight, forecast_waist = zip(*preds)
In [356]:
# graph out the weight and waist forecast
plt.figure(figsize=(20,5))
plt.plot(forecast_weight, label="Weight Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Compound Forecast Weight")
plt.legend()
plt.show()

plt.figure(figsize=(20,5))
plt.plot(forecast_waist, label="Waist Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Compound Forecast Waist")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image

Unfortunately the compounding forecast seem to be worse performing for both the entire weight and waist data.

However, it's also worth to try the compound forecast on the active lifestyle of weight and waist as well, as we've seen how well the waist prediction has done. I'll have to create the active lifestyle weight model as well.

In [292]:
# create active lifestyle weight model
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K


# Set seed for reproducibility
tf.random.set_seed(35)

# Filter training data to start post index 1150, aka where active lifestyle starts
Xweight_sequence_train_active = Xweight_sequence_train[1150:]
yweight_sequence_train_active = yweight_sequence_train[1150:]

# Ensure input shape is correct
Xweight_sequence_train_active = Xweight_sequence_train_active.reshape(Xweight_sequence_train_active.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train_active = scaler_X_weight.fit_transform(Xweight_sequence_train_active)
scaler_y_weight = StandardScaler()
normY_weight_train_active = scaler_y_weight.fit_transform(yweight_sequence_train_active.reshape(-1, 1))

# Define model
modelweight_active = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1)
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.001,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelweight_active.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.7),
                metrics=['mae'])

histweight_active = modelweight_active.fit(normX_weight_train_active, normY_weight_train_active, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [293]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))


show_scores(modelweight_active, normY_weight_test, normX_weight_test, 'Weight Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 27ms/step
Out[293]:
('Weight Evaluation',
 {'Test MAPE (%)': np.float64(203.4087975018602),
  'Test MSE': 1.7763412661267255,
  'Test R^2': -0.25398206584284244})
In [294]:
# apply simulation loop function of weight
preds = simulation_loop(modelweight_active, modelwaist_active, normX_weight_test, 
                        id_weight=0, id_waist=1, X_scaler=scaler_X_weight, 
                        scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist, 
                        start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
                        steps_ahead=len(normX_weight_test))

forecast_weight_active, forecast_waist_active = zip(*preds)
In [295]:
# visualize graph of active lifestyle models
plt.figure(figsize=(20,5))
plt.plot(forecast_weight_active, label="Weight Active Lifestyle Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Weight Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()

plt.figure(figsize=(20,5))
plt.plot(forecast_waist_active, label="Waist Active Lifestyle Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Waist Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image

It does seem the forecast prediction on active lifestyle models, give better results. However it still suffers from the same problem of over estimating values. Though for some reson, the weight active lifestyle model is the only one underperforming from the rest and is the most accurate overall.

My thoughts as to weight model, is likely because I didn't use the weight model to outfit the active lifestyle weight model. Instead, I copy pasted the waist active lifestyle over to weight, and changed the variable name.

I know for a fact that waist and active waist model were the same, however I did tweaking with the weight model instead. It seems clear that I've messed up the model for weight, and that both waist/weight data are similar enough to be able to exchange the same model, and train it separately from each other.

Trying new Quartile Loss¶

As I was finishing up my report pdf for this project, I decided to do further research on what quartile loss is and what it does. ChatGPT gave me the impression that the higher the quartile loss is, the more likely the model makes riskier predictions.

Then seeing a graph output from google image, meant that it's predicting at certain quartile levels of the y data. Aka, the higher the quartile, the more likely the model predicts at the higher bound of the range of y data. ChatGPT was technically right with stating how larger q loss makes for riskier prediction, whilst more general prediction at lower, as the predictions hovered more closer to 0.

alt text

This would then explain why the model continuously underpredicts the negative values. As the quartile loss on all my model was 0.7, with the exception of weight model (the entire dataset version), where it was 0.9. And it was evident that weight model prediction has the highest overestimation among the models.

So I'll be experimenting with quartile loss on the active models, since they make the best predictions.

In [325]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K


# Set seed for reproducibility
tf.random.set_seed(35)

# Filter training data to start post index 1150, aka where active lifestyle starts
Xwaist_sequence_train_active = Xwaist_sequence_train[1150:]
ywaist_sequence_train_active = ywaist_sequence_train[1150:]

# Ensure input shape is correct
Xwaist_sequence_train_active = Xwaist_sequence_train_active.reshape(Xwaist_sequence_train_active.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train_active = scaler_X_waist.fit_transform(Xwaist_sequence_train_active)
scaler_y_waist = StandardScaler()
normY_waist_train_active = scaler_y_waist.fit_transform(ywaist_sequence_train_active.reshape(-1, 1))

# Define model
modelwaist_active = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1)
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.001,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelwaist_active.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.7),
                metrics=['mae'])

histwaist_active = modelwaist_active.fit(normX_waist_train_active, normY_waist_train_active, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [307]:
# create active lifestyle weight model
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K


# Set seed for reproducibility
tf.random.set_seed(35)

# Filter training data to start post index 1150, aka where active lifestyle starts
Xweight_sequence_train_active = Xweight_sequence_train[1150:]
yweight_sequence_train_active = yweight_sequence_train[1150:]

# Ensure input shape is correct
Xweight_sequence_train_active = Xweight_sequence_train_active.reshape(Xweight_sequence_train_active.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train_active = scaler_X_weight.fit_transform(Xweight_sequence_train_active)
scaler_y_weight = StandardScaler()
normY_weight_train_active = scaler_y_weight.fit_transform(yweight_sequence_train_active.reshape(-1, 1))

# Define model
modelweight_active = keras.Sequential([
    layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),

    layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(256), 
    layers.BatchNormalization(),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(128),
    layers.LeakyReLU(alpha=0.1),
    layers.Dropout(0.1),

    layers.Dense(64),
    layers.LeakyReLU(alpha=0.1),
    
    layers.Dense(1)
])

lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
    initial_learning_rate=0.001,
    first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
    t_mul=2.0,  # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
    m_mul=0.8,  # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
    alpha=1e-6  # Minimum LR
)

def quantile_loss(q):
    def loss(y_true, y_pred):
        e = y_true - y_pred # calculate error difference between pred and truth
        return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
    return loss

modelweight_active.compile(optimizer = Adam(learning_rate=lr_schedule),
                loss=quantile_loss(0.65),
                metrics=['mae'])

histweight_active = modelweight_active.fit(normX_weight_train_active, normY_weight_train_active, 
                            validation_split=0.2, 
                            epochs=1000, verbose=0)

C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
In [326]:
# apply simulation loop function of weight
preds = simulation_loop(modelweight_active, modelwaist_active, normX_weight_test, 
                        id_weight=0, id_waist=1, X_scaler=scaler_X_weight, 
                        scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist, 
                        start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
                        steps_ahead=len(normX_weight_test))

forecast_weight_active, forecast_waist_active = zip(*preds)
In [357]:
# visualize graph of active lifestyle models
plt.figure(figsize=(15,5))
plt.plot(forecast_weight_active, label="Weight Active Lifestyle Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Weight Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()

plt.figure(figsize=(15,5))
plt.plot(forecast_waist_active, label="Waist Active Lifestyle Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Waist Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
No description has been provided for this image
No description has been provided for this image

Report presentations:¶

These are just some extras to include inside the report

In [328]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))


show_scores(modelweight_active, normY_weight_test, normX_weight_test, 'Weight Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
Out[328]:
('Weight Evaluation',
 {'Test MAPE (%)': np.float64(206.4920772698088),
  'Test MSE': 1.77812658512935,
  'Test R^2': -0.2552423856101027})
In [329]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))


show_scores(modelwaist_active, normY_waist_test, normX_waist_test, 'Waist Evaluation')

5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
Out[329]:
('Waist Evaluation',
 {'Test MAPE (%)': np.float64(224.53574301705336),
  'Test MSE': 1.602273135015225,
  'Test R^2': -0.18429371657718363})
In [350]:
plt.plot(histwaist_active.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[350]:
Text(0.5, 1.0, 'Loss over Epoch')
No description has been provided for this image
In [351]:
import numpy as np

# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
    preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
    return preds.mean(axis=0), preds.std(axis=0)

mean_pred_weight_active, uncertainty_active_weight = monte_carlo_predictions(modelweight_active, normX_weight_test, n_simulation=100)
In [353]:
import matplotlib.pyplot as plt

plt.figure(figsize=(15, 5))
plt.plot(range(len(mean_pred_weight_active)), mean_pred_weight_active, label="Waist Prediction", color='blue')
plt.plot(normY_weight_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_weight_active)), mean_pred_weight_active - 2 * uncertainty_active_weight, mean_pred_weight_active + 2 * uncertainty_active_weight, 
                 color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
No description has been provided for this image
In [345]:
fig, ax1 = plt.subplots(figsize=(20,5))

ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(df['Date'][1150+61:1150+61+306], df['Daily intake (kcal)'][1150+61:1150+61+306], label='Calorie consumption in kcal', color='orange', linewidth=1) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'][1150+61:1150+61+306], df['Weight (kg)'][1150+61:1150+61+306], label='Weight in kg', linewidth=1)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()
Out[345]:
<matplotlib.legend.Legend at 0x1af03dfb710>
No description has been provided for this image
In [349]:
fig, ax1 = plt.subplots(figsize=(20,5))

ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(df['Date'][700:700+306], df['Daily intake (kcal)'][700:700+306], label='Calorie consumption in kcal', color='orange', linewidth=1) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'][700:700+306], df['Weight (kg)'][700:700+306], label='Weight in kg', linewidth=1)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()
Out[349]:
<matplotlib.legend.Legend at 0x1af05731c90>
No description has been provided for this image
In [358]:
pip install nbconvert PyPDF2

Requirement already satisfied: nbconvert in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (7.16.6)
Requirement already satisfied: PyPDF2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (3.0.1)
Requirement already satisfied: beautifulsoup4 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (4.13.3)
Requirement already satisfied: bleach!=5.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach[css]!=5.0.0->nbconvert) (6.2.0)
Requirement already satisfied: defusedxml in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.7.1)
Requirement already satisfied: jinja2>=3.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.1.5)
Requirement already satisfied: jupyter-core>=4.7 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.7.2)
Requirement already satisfied: jupyterlab-pygments in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.3.0)
Requirement already satisfied: markupsafe>=2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.0.2)
Requirement already satisfied: mistune<4,>=2.0.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.1.1)
Requirement already satisfied: nbclient>=0.5.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.10.2)
Requirement already satisfied: nbformat>=5.7 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.10.4)
Requirement already satisfied: packaging in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (24.2)
Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (1.5.1)
Requirement already satisfied: pygments>=2.4.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (2.19.1)
Requirement already satisfied: traitlets>=5.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.14.3)
Requirement already satisfied: webencodings in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach!=5.0.0->bleach[css]!=5.0.0->nbconvert) (0.5.1)
Requirement already satisfied: tinycss2<1.5,>=1.1.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach[css]!=5.0.0->nbconvert) (1.4.0)
Requirement already satisfied: platformdirs>=2.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-core>=4.7->nbconvert) (4.3.6)
Requirement already satisfied: pywin32>=300 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-core>=4.7->nbconvert) (308)
Requirement already satisfied: jupyter-client>=6.1.12 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbclient>=0.5.0->nbconvert) (8.6.3)
Requirement already satisfied: fastjsonschema>=2.15 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbformat>=5.7->nbconvert) (2.21.1)
Requirement already satisfied: jsonschema>=2.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbformat>=5.7->nbconvert) (4.23.0)
Requirement already satisfied: soupsieve>1.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from beautifulsoup4->nbconvert) (2.6)
Requirement already satisfied: typing-extensions>=4.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from beautifulsoup4->nbconvert) (4.12.2)
Requirement already satisfied: attrs>=22.2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (25.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (2024.10.1)
Requirement already satisfied: referencing>=0.28.4 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (0.36.2)
Requirement already satisfied: rpds-py>=0.7.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (0.22.3)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (2.9.0.post0)
Requirement already satisfied: pyzmq>=23.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (26.2.1)
Requirement already satisfied: tornado>=6.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (6.4.2)
Requirement already satisfied: six>=1.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (1.17.0)
Note: you may need to restart the kernel to use updated packages.

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip
In [ ]: